Sigma-Ma commented on PR #174:
URL: https://github.com/apache/hbase-connectors/pull/174#issuecomment-6872096842

   > > While e2e testing these, I run into a situation where a query with IS 
NULL predicate still returned some rows where the given column wasn't null.
   > > Was checking this with Claude. the cause seems to be the due to 
StoreScanner returning mid-rows once hbase.storescanner.pread.max.bytes limit 
(256KB) is reached. This seems to cause the SparkSQLPushDownFilter.filterRow 
being called only on the second portion of the row, but when the cell we should 
check for null is in the first portion, the row gets in.
   > > I've checked disabling pread by setting 
hbase.storescanner.pread.max.bytes to -1, or also setting it to 1GB, and query 
works.
   > 
   > Suggested fix by claude:
   > 
   > 1. Override hasFilterRow to always return true in SparkSQLPushDownFilter 
(similar to hbase builtin filter SingleColumnValueFilter), so that 
RegionScannerImpl then moves the size and time limits to BETWEEN_ROWS;
   > 2. Only call setBatch when no pushdown filter is attached. 
HBasePartitionReader.buildScanner (line 180):
   > 
   > ` if (batch > 0 && !hasNullCheck && pushDownFilter.isEmpty) 
scan.setBatch(batch)`
   
   Thanks, that was the missing bit. I’ve added hasFilterRow() to the shared 
filter and guarded setBatch in both Spark 3 and 4. I also added a regression 
with flushed wide rows and a low PREAD threshold, checking NULLs near both ends 
of the row.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to