keksmd opened a new pull request, #975: URL: https://github.com/apache/incubator-graphar/pull/975
### Reason for this PR Closes #964. #961 replaced the row-oriented physical batch contract with `ValueVector` / `VectorRecordBatch`; this is the Parquet backend for that contract. It is the physical IO layer the pure-Java reader and writer (#947) sit on, and it keeps the layout-free boundary: no GraphAr semantics, no Arrow, no `Row` facade. ### What changes are included in this PR? New module `graphar-io-parquet` (wired into `maven-projects/pom.xml` and the Java SDK workflow): - `ParquetPhysicalReader` / `ParquetBatchCursor`: reads one row group per batch, applies physical projection and the strict half-open row range, and reports `PROJECTION` / `ROW_RANGE` as applied. Each batch is a `VectorRecordBatch` of `ParquetValueVector` columns. - `ParquetValueVector`: binary values come back as read-only `ByteBuffer`s over a private copy and lists as immutable `List`s, so a consumer never aliases the decoder's buffers. - `ParquetPhysicalWriter`: emits Offset Indexes with bounded pages. `WriteMode.APPEND` is rejected with `UnsupportedOperationException` because Parquet cannot append to a closed file; it is never treated as an overwrite. - `io-api`: `ObjectValueVector` (in-memory vector) and `RecordBatches.ofRows` / `ofArrays` (row-to-column transposer) so producers can build column batches without Arrow. Non-goals kept out: GraphAr reader/writer/validator consumers, Arrow, changes to the decoding algorithm. ### Are these changes tested? Yes, from `maven-projects/` with the same commands the workflow runs: ``` mvn --no-transfer-progress -pl storage-api,storage-local,io-api,io-parquet,storage-s3,core -am spotless:check mvn --no-transfer-progress -pl storage-api,storage-local,io-api,io-parquet,storage-s3,core -am clean verify -Dspotless.check.skip=true mvn --no-transfer-progress -pl io-api,io-parquet -am package javadoc:javadoc -DskipTests -Dspotless.check.skip=true ``` All green: io-api 79 tests, io-parquet 8 tests. - `RecordBatchesTest`: schema position, row/column counts, nulls, width mismatch rejection, list snapshot on read. - `ParquetPhysicalReaderFixtureTest`: canonical `ldbc_sample` fixture; projection and row-range reads retain applied `PROJECTION` and `ROW_RANGE`. - `ParquetListRoundTripTest`: nullable string lists round-trip, empty and null preserved. - `ParquetPhysicalWriterOffsetIndexTest`: written files carry readable Offset Indexes with the bounded-page assertion. - `ParquetPhysicalWriterModeTest`: `APPEND` throws and leaves the existing file byte-identical; `OVERWRITE` replaces it. - `ParquetPhysicalIoEfficiencyTest`: projection and row range only touch the needed column chunks / row groups. `pre-commit run --files` on the changed paths: passed. ### Are there any user-facing changes? New public module `org.apache.graphar:graphar-io-parquet` and two new public io-api classes (`ObjectValueVector`, `RecordBatches`). No changes to existing public APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
