9aman commented on code in PR #20168:
URL: https://github.com/apache/pinot/pull/20168#discussion_r4522336589
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -608,6 +608,17 @@ static void validateIngestionConfig(TableConfig
tableConfig, Schema schema,
} else {
LOGGER.warn("It's not recommended to create pauseless tables with
replication 1 for stability reasons.");
}
+ // Pauseless consumption needs a row or size flush threshold, which
stops all replicas at the same offset. Each
Review Comment:
Thanks. Changes in 0bb8283831:
- The comment no longer says that a row or size threshold alone stops all
replicas at the same offset. It says that the check rejects only an explicit
time threshold, and that the default threshold still applies.
- The check rejects large values such as `7d` and an explicit `6h` on
purpose. Each replica checks any time threshold on its own clock, so it can
stop replicas at different offsets. For pauseless tables, use a row or size
threshold.
- The 6-hour default needs a change on the consumption side. It is also the
only limit on the age of a consuming segment, so removing it can let a slow
partition outlive the stream retention. A separate PR will handle it.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -608,6 +608,17 @@ static void validateIngestionConfig(TableConfig
tableConfig, Schema schema,
} else {
LOGGER.warn("It's not recommended to create pauseless tables with
replication 1 for stability reasons.");
}
+ // Pauseless consumption needs a row or size flush threshold, which
stops all replicas at the same offset. Each
+ // replica checks a time threshold on its own (a restarted replica
also extends it), so replicas can stop at
+ // different offsets. A replica past the committed end offset can only
download the segment, so if the
+ // committer fails before the upload, no replica has a copy.
+ for (Map<String, String> streamConfigMap : streamConfigMaps) {
+ for (String key :
List.of(StreamConfigProperties.SEGMENT_FLUSH_THRESHOLD_TIME,
+ StreamConfigProperties.SEGMENT_FLUSH_THRESHOLD_TIME +
StreamConfigProperties.LLC_SUFFIX)) {
+ Preconditions.checkState(!streamConfigMap.containsKey(key),
Review Comment:
Good catch, and thanks for the repro. Fixed in 0bb8283831. The check now
allows the key when the stored config is already pauseless and has the same
value, like `hasSameTransformConfig` does for volatile transforms. A schema
update passes the stored config as `existingTableConfig`, so it continues to
work. The check still rejects a new table with the key, a changed value, and
enabling pauseless consumption on a table with the key.
`pauselessTimeFlushThresholdTest` covers these cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]