CalvinKirs opened a new pull request, #61269:
URL: https://github.com/apache/doris/pull/61269
This PR unifies the FE-side OSS Hadoop filesystem implementation to Jindo FS
and removes legacy OSS filesystem dependencies that are no longer needed.
## Why
We currently have multiple OSS filesystem implementations on the FE
classpath, including:
- `org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem`
- `paimon-oss`
This makes OSS behavior inconsistent and increases the chance of classpath
conflicts. Since Doris already packages and uses Jindo FS, FE should
consistently use Jindo instead of mixing multiple OSS filesystem
implementations.
## Changes
- Switch `OSSProperties` to use Jindo FS:
- `fs.oss.impl = com.aliyun.jindodata.oss.JindoOssFileSystem`
- `fs.AbstractFileSystem.oss.impl = com.aliyun.jindodata.oss.JindoOSS`
- Keep `OSSHdfsProperties` aligned with the same Jindo FS constants.
- Add FE unit test coverage to verify OSS Hadoop config is initialized with
Jindo FS.
- Remove legacy OSS filesystem dependencies from FE modules:
- remove `paimon-oss` from `fe-core`
- remove `paimon-oss` from `preload-extensions`
- remove `hadoop-aliyun` from FE dependency management and `hadoop-deps`
## Scope
This PR only updates FE-side OSS filesystem wiring and FE-related dependency
cleanup.
Non-FE modules are intentionally left unchanged.
## Verification
- `run-fe-ut.sh --run
org.apache.doris.datasource.property.storage.OSSPropertiesTest,org.apache.doris.datasource.property.storage.OSSHdfsPropertiesTest`
- Full FE reactor build passed
## Notes
`aliyun-sdk-oss` is still kept because it is still used by FE cloud storage
code (`OssRemote`) and is not part of the Hadoop OSS filesystem implementation
cleanup in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
