Sigma-Ma opened a new pull request, #175: URL: https://github.com/apache/hbase-connectors/pull/175
### What changes were proposed in this pull request? Read all visible cells in Spark 4 Scan and Get operations, then project the requested fields when constructing Spark rows. This preserves sparse rows whose projected qualifiers are absent. Disable cell batching so each HBase Result represents a complete Spark row, and document that the Spark 4 datasource ignores `hbase.spark.query.batchsize`. Add MiniCluster regression tests covering sparse rows, unmapped qualifiers and column families, wide rows, batched Gets, predicate pushdown enabled and disabled, and timestamp/time-range reads. ### Why are the changes needed? Restricting Scan and Get operations to projected qualifiers can remove an entire row when those qualifiers are missing, even though the row has visible cells in other columns. For example, given `r1: cf:name=Alice` and `r2: cf:age=20`, selecting `key, name` should return both `(r1, Alice)` and `(r2, null)`. The current implementation can omit r2. Cell batching can also split a wide row into multiple Results, which the reader currently converts into separate Spark rows. Reading complete rows increases I/O for wide tables, but preserves row existence independently of the projection and predicate pushdown settings. ### How was this patch tested? ```bash mvn -B -ntp -Pspark4 -pl spark4/hbase-spark4 -Dsuites=org.apache.hadoop.hbase.spark.datasources.HBaseTableProviderSuite '-Dhbase-surefire.jdk17.flags=-Xmx2g -XX:ActiveProcessorCount=4' test ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
