wchevreuil commented on PR #174:
URL: https://github.com/apache/hbase-connectors/pull/174#issuecomment-6836566152
> While e2e testing these, I run into a situation where a query with IS NULL
predicate still returned some rows where the given column wasn't null.
>
> Was checking this with Claude. the cause seems to be the due to
StoreScanner returning mid-rows once hbase.storescanner.pread.max.bytes limit
(256KB) is reached. This seems to cause the SparkSQLPushDownFilter.filterRow
being called only on the second portion of the row, but when the cell we should
check for null is in the first portion, the row gets in.
>
> I've checked disabling pread by setting hbase.storescanner.pread.max.bytes
to -1, or also setting it to 1GB, and query works.
Suggested fix by claude:
1. Override hasFilterRow to always return true in SparkSQLPushDownFilter
(similar to hbase builtin filter SingleColumnValueFilter), so that
RegionScannerImpl then moves the size and time limits to BETWEEN_ROWS;
2. Only call setBatch when no pushdown filter is attached.
HBasePartitionReader.buildScanner (line 180):
` if (batch > 0 && !hasNullCheck && pushDownFilter.isEmpty)
scan.setBatch(batch)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
