JingsongLi commented on code in PR #8380:
URL: https://github.com/apache/paimon/pull/8380#discussion_r3492863359
##########
paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java:
##########
@@ -109,6 +117,10 @@ public DataEvolutionCompactCoordinator(
}
public List<DataEvolutionCompactTask> plan() {
+ if (deletionVectorsEnabled) {
+ return Collections.emptyList();
Review Comment:
Returning an empty task list here leaves
`DataEvolutionTableCompactSource.pollNext` returning `MORE_AVAILABLE` forever,
because that source only ends on `EndOfScanException`. For DV-enabled
data-evolution tables, compact actions can spin without making progress. Please
fail fast/throw an end signal, or have the source translate this case to
`END_OF_INPUT`, and add a regression test covering compact source behavior with
`data-evolution.enabled=true` and `deletion-vectors.enabled=true`.
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionFileStoreScan.java:
##########
@@ -62,12 +62,14 @@
import static org.apache.paimon.format.blob.BlobFileFormat.isBlobFile;
import static org.apache.paimon.manifest.ManifestFileMeta.allContainsRowId;
import static org.apache.paimon.types.VectorType.isVectorStoreFile;
+import static org.apache.paimon.utils.DataEvolutionUtils.retrieveAnchorFile;
/** {@link FileStoreScan} for data-evolution enabled table. */
public class DataEvolutionFileStoreScan extends AppendOnlyFileStoreScan {
private boolean dropStats = false;
@Nullable private RowType readType;
+ private final boolean deletionVectorsEnabled;
Review Comment:
This new DV flag also needs to guard the manifest-level limit pushdown in
`readManifestEntries`. Today that code still counts the raw row-range size and
may stop after a range whose rows are deleted by DV; for example, if the first
5-row group is fully deleted, `LIMIT 1` can plan only that group and the reader
returns 0 rows. Please either disable that optimization when deletion vectors
are enabled or make it DV-cardinality aware, and add a regression test for
`withLimit` where early row-id groups contain deleted rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
