JingsongLi commented on code in PR #9534:
URL: https://github.com/apache/paimon/pull/9534#discussion_r3911739880
##########
paimon-format/src/main/java/org/apache/orc/impl/WriterImpl.java:
##########
@@ -321,9 +324,12 @@ public boolean checkMemory(double newScale) throws
IOException {
}
private boolean checkMemory() throws IOException {
- if (rowsSinceCheck >= rowsPerCheck) {
+ long size =
+ rowsSinceCheck < rowsPerCheck && stripeSizePerCheck == 0
+ ? 0
+ : treeWriter.estimateMemory();
+ if (rowsSinceCheck >= rowsPerCheck || size > stripeSizePerCheck) {
Review Comment:
[P2] Honor sub-1 stripe-size ratios
This new gate says a stripe is checked once `estimateMemory()` exceeds
`ratio * orc.stripe.size`, but the unchanged flush condition below still
compares only with `memoryLimit` (normally the full stripe size). For example,
with ratio `0.5` and an unscaled writer, this branch starts firing at half a
stripe, resets `rowsSinceCheck` on every subsequent batch, yet does not flush
until the estimate exceeds the full stripe size. That both violates the option
description and adds an estimate on every batch in between. Please either flush
against the configured threshold (while still respecting a lower MemoryManager
limit), or reject/document ratios below 1 if they are not supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
