9aman commented on code in PR #20168:
URL: https://github.com/apache/pinot/pull/20168#discussion_r4522336589


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -608,6 +608,17 @@ static void validateIngestionConfig(TableConfig 
tableConfig, Schema schema,
         } else {
           LOGGER.warn("It's not recommended to create pauseless tables with 
replication 1 for stability reasons.");
         }
+        // Pauseless consumption needs a row or size flush threshold, which 
stops all replicas at the same offset. Each

Review Comment:
   Thanks. Changes in 0bb8283831:
   - The comment no longer says that a row or size threshold alone stops all 
replicas at the same offset. It says that the check rejects only an explicit 
time threshold, and that the default threshold still applies.
   - The check rejects large values such as `7d` and an explicit `6h` on 
purpose. Each replica checks any time threshold on its own clock, so it can 
stop replicas at different offsets. For pauseless tables, use a row or size 
threshold.
   - The 6-hour default needs a change on the consumption side. It is also the 
only limit on the age of a consuming segment, so removing it can let a slow 
partition outlive the stream retention. A separate PR will handle it.
   



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -608,6 +608,17 @@ static void validateIngestionConfig(TableConfig 
tableConfig, Schema schema,
         } else {
           LOGGER.warn("It's not recommended to create pauseless tables with 
replication 1 for stability reasons.");
         }
+        // Pauseless consumption needs a row or size flush threshold, which 
stops all replicas at the same offset. Each
+        // replica checks a time threshold on its own (a restarted replica 
also extends it), so replicas can stop at
+        // different offsets. A replica past the committed end offset can only 
download the segment, so if the
+        // committer fails before the upload, no replica has a copy.
+        for (Map<String, String> streamConfigMap : streamConfigMaps) {
+          for (String key : 
List.of(StreamConfigProperties.SEGMENT_FLUSH_THRESHOLD_TIME,
+              StreamConfigProperties.SEGMENT_FLUSH_THRESHOLD_TIME + 
StreamConfigProperties.LLC_SUFFIX)) {
+            Preconditions.checkState(!streamConfigMap.containsKey(key),

Review Comment:
   Good catch, and thanks for the repro. Fixed in 0bb8283831. The check now 
allows the key when the stored config is already pauseless and has the same 
value, like `hasSameTransformConfig` does for volatile transforms. A schema 
update passes the stored config as `existingTableConfig`, so it continues to 
work. The check still rejects a new table with the key, a changed value, and 
enabling pauseless consumption on a table with the key. 
`pauselessTimeFlushThresholdTest` covers these cases.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to