keksmd commented on PR #975: URL: https://github.com/apache/incubator-graphar/pull/975#issuecomment-5697387922
Yes, considered, and I share the concern about the Hadoop tail. Current state and reasoning: **What this PR actually depends on.** Our code has zero `org.apache.hadoop` imports. GraphAr storage (`storage-api`: `InputFile`/`SeekableInput`/`PositionOutput`) is what Parquet reads from and writes to, via `ParquetInputFile`/`ParquetOutputFile` adapters — no S3A, no Hadoop `FileSystem`. `hadoop-common` and `hadoop-mapreduce-client-core` are there only because `parquet-hadoop` needs `Configuration` and MapReduce input classes on the classpath at runtime. So the Hadoop dependency is an accident of `parquet-java` packaging, not a design choice. **Why `parquet-java` for the first slice.** - Java 11 baseline of the SDK. Hardwood requires Java 21 (`maven.compiler.release=21` in its pom; native build on 25). - Write to object storage: Hardwood's writer targets `OutputFile.of(Path)` only, object-store output is "coming soon" (README, 1.1.0.Beta1, 2026-08-31). Our writer goes through `storage-api`, S3 included, today. - Maturity as a format oracle: the C++ implementation is the reference, `parquet-java` is the canonical Java counterpart. For a first columnar slice I preferred boring-and-correct. **What is already backend-neutral.** `io-api` (`PhysicalReader`/`PhysicalWriter`, `RecordBatch`/`ValueVector`, `ReadRequest` with projection/range/filter/limit and the applied-hints report) has no Parquet or Hadoop types. `io-parquet` is one implementation behind it; `reader`/`writer` never see `parquet-java`. Swapping or adding an engine is a module, not an API change. **What I would propose.** 1. Keep this PR as the `parquet-java` backend, but I can shrink the tail right here: replace `hadoop-common` + `hadoop-mapreduce-client-core` with the shaded `hadoop-client-api` (single jar, no transitive Guava/Jackson/etc.). If you want that in this PR, say so and I will do it. 2. Follow-up issue: a Hardwood backend spike as a separate optional module, Java 21, behind the same `PhysicalReader`/`PhysicalWriter`, with a JMH comparison on topology (INT64 src/dst) and property chunks. Hardwood's typed `ColumnReader`/`ColumnBatch` maps well onto column vectors and would let us add primitive accessors to `ValueVector` without boxing; I would rather do that with numbers than by assumption. 3. Decision needed from maintainers: is Java 21 acceptable for an optional module while the core SDK stays on 11? If yes, Hardwood is a real candidate for the default once its object-store write lands. If no, `parquet-java` + shaded Hadoop client is the realistic option. Happy to open the follow-up issue if this direction works for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
