Sigma-Ma opened a new pull request, #174: URL: https://github.com/apache/hbase-connectors/pull/174
### What changes were proposed in this pull request? Add shared predicate capability checks to the Spark 3 and Spark 4 connectors. Leave unsupported comparisons to Spark and exclude them from server-side filtering and row-key pruning. Return compound AND/OR predicates intact when a child is unsupported. This is a draft. Two Spark 4 review findings remain unresolved: residual NULL checks can lose sparse rows, and primitive Avro equality fallback can raise a ClassCastException. ### Why are the changes needed? Some accepted predicates use encodings or comparison semantics that do not match the stored HBase values. They can return incorrect results or throw exceptions. Spark residual filtering cannot recover rows already excluded by HBase. ### How was this patch tested? ```bash export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-17.0.2.jdk/Contents/Home mvn -B -ntp -Pspark4 -pl spark/hbase-spark,spark4/hbase-spark4 -am -DskipTests install mvn -B -ntp -nsu -pl spark/hbase-spark -Dtest=NoJavaTests -Dsurefire.failIfNoSpecifiedTests=false -Dsuites=org.apache.hadoop.hbase.spark.DefaultSourceSuite,org.apache.hadoop.hbase.spark.PartitionFilterSuite,org.apache.hadoop.hbase.spark.RowKeyFilterSuite,org.apache.hadoop.hbase.spark.datasources.NaiveEncoderSuite -Dsurefire.rerunFailingTestsCount=0 test mvn -B -ntp -nsu -Pspark4 -pl spark4/hbase-spark4 -Dtest=NoJavaTests -Dsurefire.failIfNoSpecifiedTests=false -Dsuites=org.apache.hadoop.hbase.spark.datasources.HBaseScanBuilderSuite,org.apache.hadoop.hbase.spark.datasources.HBaseTableProviderSuite,org.apache.hadoop.hbase.spark.datasources.RowKeyFilterSuite,org.apache.hadoop.hbase.spark.datasources.ScanRangeSuite,org.apache.hadoop.hbase.spark.datasources.NaiveEncoderSuite -Dsurefire.rerunFailingTestsCount=0 test mvn -B -ntp -nsu -Pspark4 -pl spark/hbase-spark-pushdown,spark/hbase-spark,spark4/hbase-spark4 '-DspotlessFiles=.*(PredicatePushdownSupport|DefaultSource|DefaultSourceSuite|HBaseScanBuilder|HBaseScanBuilderSuite|HBaseTableProviderSuite)[.]scala' spotless:check ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
