yihua opened a new pull request, #6785:
URL: https://github.com/apache/hudi/pull/6785
### Change Logs
When using the HoodieSnapshotExporter to export a Hudi dataset on S3 to a
different bucket, i.e., the source-base-path and the target-output-path are in
different buckets, IllegalArgumentException is thrown:
```
./bin/spark-submit \
...
--jars /home/hadoop/hudi-spark3.2-bundle_2.12-0.13.0-SNAPSHOT.jar \
--class "org.apache.hudi.utilities.HoodieSnapshotExporter" \
/home/hadoop/hudi-utilities-bundle_2.12-0.13.0-SNAPSHOT.jar \
--source-base-path "s3a://ethan-lakehouse-us-east-2/hudi/hudi_trips_cow/" \
--target-output-path "s3a://ethan-tmp/backup/" \
--output-format "hudi"
```
```
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS
s3a://ethan-tmp//backup -expected s3a://ethan-lakehouse-us-east-2
at
org.apache.hadoop.fs.s3native.S3xLoginHelper.checkPath(S3xLoginHelper.java:224)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.checkPath(S3AFileSystem.java:1155)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:666)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.makeQualified(S3AFileSystem.java:1117)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.qualify(S3AFileSystem.java:1143)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3078)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4263)
at
org.apache.hudi.utilities.HoodieSnapshotExporter.outputPathExists(HoodieSnapshotExporter.java:145)
at
org.apache.hudi.utilities.HoodieSnapshotExporter.export(HoodieSnapshotExporter.java:120)
at
org.apache.hudi.utilities.HoodieSnapshotExporter.main(HoodieSnapshotExporter.java:275)
```
The root cause is that, when copying the data from the source path to the
target path, the same file system based on the source path is used for writing
the destination file, causing the exception.
This PR fixes the problem by using the correct file system.
### Impact
**Risk level: none**
The PR is tested on EMR 6.7.0 with OSS Spark 3.2.2. Exporting the dataset
to a different S3 bucket in "hudi" or "parquet" format is successful.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
