Sigma-Ma opened a new pull request, #177:
URL: https://github.com/apache/hbase-connectors/pull/177

   JIRA: https://issues.apache.org/jira/browse/HBASE-30513
   
   ## What changes were proposed in this pull request?
   
   Restore datasource option > SparkConf > default precedence for these Spark 4 
read settings:
   
   - `hbase.spark.query.cacheblocks`
   - `hbase.spark.query.cachedrows`
   - `hbase.spark.bulkget.size`
   
   Resolve these settings on the driver and pass the immutable values to 
partition readers through the serialized reader factory.
   
   Keep `hbase.spark.query.batchsize` behavior unchanged: per-read options 
still control cell batching, while the SparkConf value remains ignored.
   
   ## Why are the changes needed?
   
   Spark 4 currently reads these settings only from datasource options. 
Applications that configure them globally through SparkConf can silently use 
different read settings after upgrading from Spark 3.
   
   Restoring the existing precedence preserves global settings while allowing 
per-read overrides.
   
   ## How was this patch tested?
   
   ```bash
   
   mvn -B -ntp -Pspark4 -pl spark4/hbase-spark4 -am -DskipTests clean install
   
   mvn -B -ntp -Pspark4 -pl spark4/hbase-spark4 
-Dsuites=org.apache.hadoop.hbase.spark.datasources.HBaseReadOptionsSuite,org.apache.hadoop.hbase.spark.datasources.HBaseReadOptionsReaderSuite,org.apache.hadoop.hbase.spark.datasources.HBaseScanBuilderSuite,org.apache.hadoop.hbase.spark.datasources.HBaseTableProviderSuite
 -Dsurefire.rerunFailingTestsCount=0 test
   ```
   
   The new tests cover configuration precedence, defaults, invalid values, 
factory serialization, actual Scan parameters and Get batch sizes. They also 
verify that existing batchsize behavior is preserved for Scan and Get-only 
reads.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to