guiyanakuang commented on code in PR #1118:
URL: https://github.com/apache/orc/pull/1118#discussion_r876535187
##########
java/core/src/java/org/apache/orc/OrcConf.java:
##########
@@ -29,6 +29,8 @@ public enum OrcConf {
STRIPE_SIZE("orc.stripe.size", "hive.exec.orc.default.stripe.size",
64L * 1024 * 1024,
"Define the default ORC stripe size, in bytes."),
+ STRIPE_ROW_COUNT("orc.stripe.row.count","orc.stripe.row.count",
+ Integer.MAX_VALUE,"This value limit the row count in one stripe."),
Review Comment:
Keep a space after the comma.
Agree with setting the default value to Integer.MAX_VALUE, so that the
default config is compatible with future user upgrades and has no additional
impact.
##########
java/core/src/java/org/apache/orc/impl/WriterImpl.java:
##########
@@ -300,7 +303,7 @@ private boolean checkMemory() throws IOException {
LOG.debug("ORC writer " + physicalWriter + " size = " + size +
" limit = " + memoryLimit);
}
- if (size > memoryLimit) {
+ if (size > memoryLimit || rowsInStripe > stripeRowCount) {
Review Comment:
Adding rowsInStripe and stripeRowCount to the debug print in the above lines
would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
