HyukjinKwon commented on PR #59313:
URL: https://github.com/apache/spark/pull/59313#issuecomment-6139365807
@viirya Thanks for the review from the library author's side. I addressed
all 7 inline comments in ff7b1ae0150f and replied to each of them. On the two
other points:
- **Cluster-mode test**: added `NativeDataSourceClusterSuite`, in
`local-cluster[2,1,1024]`:
- "read and write with a package added as an artifact": a batch read over
4 partitions, so that both executor JVMs load the library, a batch write, and a
streaming read and write over several micro-batches.
- "executors load the distributed copy of a configured package": a package
under `spark.sql.dataSource.native.paths` is prepared on the driver (read,
loaded, and copied to the artifacts of the session as `<fileName>.<sha256>`).
Then the file at the driver's path is overwritten with zeros of the same length
and modification time, so the driver keeps its cached package but the executors
cannot use that file. The query only succeeds if the executors find the
checksum-named copy under the session UUID in `distributedCopies` and pass the
SHA-256 check. I checked that it catches a regression: with `locate` limited to
the driver's path, it fails with `NATIVE_DATA_SOURCE_PACKAGE_NOT_FOUND`.
- **Statistics**: agreed that it can be a later optional function. The docs
page now has a "Limitations" section for what version 1 doesn't cover yet:
statistics (a native scan is sized at `spark.sql.defaultSizeInBytes`, so it is
never the broadcast side of a join without a hint), custom metrics, admission
control for streaming reads and continuous streaming, the pushdown of
aggregates, sort order and partitioning, and cancelling a read in progress.
@gerashegalov Thanks, fixed: `file://` URIs are now passed to the JVM as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
