ZZZxDong commented on code in PR #8837:
URL: https://github.com/apache/paimon/pull/8837#discussion_r3710796633
##########
docs/generated/spark_connector_configuration.html:
##########
@@ -26,6 +26,12 @@
</tr>
</thead>
<tbody>
+ <tr>
+ <td><h5>delete.point-delete.max-rows</h5></td>
Review Comment:
Yes — the reason this option existed (driver OOM while materializing the
keys) is gone now that the literal path cross joins one small DataFrame per pk
column, so the driver only holds the literals of the condition itself.
One consequence worth deciding on before I drop the bound completely:
without any limit, the fast path writes the whole cartesian product no matter
whether those keys exist. A single pk column is inherently bounded by the IN
list, but a composite pk with several multi-value lists is not:
```sql
DELETE FROM t WHERE a IN (1000 values) AND b IN (1000 values) AND c IN (1000
values);
```
That is a small query that writes 10^9 -D records. It does not fail, it just
bloats L0, does the same number of lookups on a deletion-vectors table, and can
materialize partitions that never existed. The scan path would only write as
many -D rows as there are matching rows in the table.
Three ways to go, happy with any:
1. Drop it entirely — simplest, and the case above needs a fairly deliberate
query.
2. Keep the bound as an internal constant, no user facing option.
3. No option and no magic number: fall back when the product exceeds the
latest snapshot's `totalRecordCount`, i.e. when we would write more -D rows
than the table has rows — exactly the point where scanning is cheaper anyway.
I slightly prefer 3, but if you think the multi-column case is too contrived
I will just remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
