malinjawi opened a new pull request, #13386: URL: https://github.com/apache/gluten/pull/13386
## What changes are proposed in this pull request? The Velox Parquet writer only opens files on local, HDFS, S3 (`s3`, `s3a`, `oss`, `cos`, `cosn`), GCS and ABFS paths (`VeloxRuntime::createDataSource`). For any other scheme it throws "The file path is not local when writing data with parquet format in velox runtime!". On Spark 4.x, `GlutenDeltaNativeWriteGate` never looked at the table path, so a native Delta write to, for example, `viewfs`, `wasbs` or a custom Hadoop `FileSystem` failed the task instead of falling back. This checks the data path's scheme before the native write and uses Delta's writer for the rest. The fallback list in the docs gets the new entry. Spark 3.5 has no native write gate yet (#13383 adds the first one). The same check can go there once it lands. ## How was this patch tested? - New case in `GlutenDeltaNativeWriteGateSuite`; the suite passes locally on Spark 4.1 (19/19). - Delta Lake 4.4's `DeletionVectorWriteDurabilitySuite` writes through a custom filesystem. Its five tests failed with the error above, so they leave the known-failures baseline here, and the Delta UT gate checks they now pass. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: IBM Bob -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
