Shekharrajak opened a new pull request, #3068: URL: https://github.com/apache/datafusion-comet/pull/3068
## Which issue does this PR close? Closes #3015. ## Rationale for this change The native Parquet writer needed a fix to use `output_path` as the base directory for file writes when `work_dir` is not set. Without this fix, files were being written to root (`/`) instead of the intended output directory. ## What changes are included in this PR? 1. **Protobuf**: Added `staging_file_path` field to `ParquetWriter` message for future 2PC support 2. **Native Rust**: Fixed parquet_writer.rs to use `output_path` as fallback when `work_dir` is empty 3. **Scala/JVM**: Simplified `CometNativeWriteExec` to write directly to output path 4. **Tests**: Added `CometParquetWriter2PCSuite` with basic write functionality tests ## How are these changes tested? Added `CometParquetWriter2PCSuite` with 5 tests: - Basic successful write creates files in output directory - Multiple concurrent tasks write without file conflicts - Various data types write correctly - Overwrite mode replaces existing files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
