nanxiuzi opened a new pull request, #18633: URL: https://github.com/apache/dolphinscheduler/pull/18633
<!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! --> ## Was this PR generated or assisted by AI? YES - AI assisted. The issue analysis, the fix proposal and the code change were drafted with AI assistance; I reviewed the change, built the module locally and validated it against a real OSS bucket. ## Purpose of the pull request Close #18632. When the resource center uses the OSS storage backend, `OssStorageOperator#listStorageEntity` issued a single `listObjectsV2` call with no pagination. The OSS server default of 100 `max-keys` therefore applied, and because `Contents` (objects) and `CommonPrefixes` (directories) share that one quota, any directory with more than 100 combined entries was silently truncated in the UI. Widening the page cannot fix this on its own: `max-keys` is capped at 1000 by the service. ## Brief change log - `OssStorageOperator#listStorageEntity` now loops on `IsTruncated` / `NextContinuationToken` until all pages are fetched, setting `max-keys` to 1000 per request (`MAX_KEYS` constant). - Common prefixes are deduplicated with a `Set` that persists across pages, so directory marker objects are still filtered out when they appear on a later page than their common prefix. - The current directory itself and directory markers already present in `CommonPrefixes` continue to be excluded from the object listing. ## Verify this pull request This change was verified as follows: - `mvn -pl dolphinscheduler-storage-plugin/dolphinscheduler-storage-oss -am -DskipTests install` builds successfully, and `spotless:apply` leaves the file unchanged. - Manually verified against a real Aliyun OSS bucket: a resource directory containing more than 100 mixed files and sub-directories now lists every entry in the resource center, where before only the first 100 were returned. Nested listing via `listFileStorageEntityRecursively` is likewise complete. - No existing automated tests cover this class (`dolphinscheduler-storage-oss` has no `src/test` directory), so no tests were added. A mocked-SDK unit test could be added if maintainers want one. ## Pull Request Notice [Pull Request Notice](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md) If your pull request contains incompatible change, you should also add it to `docs/docs/en/guide/upgrade/incompatible.md` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
