JingsongLi opened a new pull request, #8615: URL: https://github.com/apache/paimon/pull/8615
## Summary Complete source-backed primary-key BTree and Bitmap support by enabling postpone-bucket batch pruning, adding Flink and Spark end-to-end coverage, and documenting configuration and fallback behavior. ## Changes - Allow `PrimaryKeyBatchScan` to apply scalar indexes to real-bucket splits of postpone-bucket tables. - Add a regression test proving postpone-bucket scans return `IndexedSplit` when a snapshot-scoped sorted index is available. - Exercise mixed BTree and Bitmap maintenance and ordinary scan pruning through Flink and Spark SQL, including updates, deletes, `AND`/`OR` predicates, partial coverage, and unindexed fallback. - Document column-list configuration, field options, fixed/postpone bucket requirements, asynchronous coverage, residual filtering, schema evolution, and current limitations. - Cross-link scalar indexes from the vector-index documentation. ## Testing - [x] `mvn -pl paimon-codegen-loader -am -Pfast-build -DskipTests package` - [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=PrimaryKeySortedIndexBatchScanTest test` - [x] `mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build -Pflink1 -DfailIfNoTests=false -DwildcardSuites=none -Dtest=PrimaryKeySortedIndexITCase test` - [x] `JAVA_HOME=/Users/lijingsong/Library/Java/JavaVirtualMachines/corretto-11.0.25/Contents/Home mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build -Pflink2 -DfailIfNoTests=false -DwildcardSuites=none -Dtest=PrimaryKeySortedIndexITCase test` - [x] `mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -Pspark3 -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.PrimaryKeySortedIndexTest -Dtest=none test` - [x] `mvn package -Pgenerate-docs -pl paimon-docs -nsu -DskipTests -am` - [x] `mvn -pl paimon-core -am -DskipTests validate` - [x] `mvn -pl paimon-flink/paimon-flink-common -Pflink1 -DskipTests validate` - [x] `git diff --check` ## Notes The targeted connector tests require `paimon-codegen-loader` resources produced during `prepare-package`, so the loader package step was run before the test commands. This follows the batch-scan integration merged in #8603. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
