Copilot commented on code in PR #19244:
URL: https://github.com/apache/pinot/pull/19244#discussion_r3773696484
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -620,7 +621,9 @@ static void validateIngestionConfig(TableConfig
tableConfig, Schema schema,
+ columnName + "'");
}
try {
- validateIngestionTransformFunctionVolatility(transformConfig,
existingTransformConfigs);
+ if (tableConfig.getTableType() == TableType.REALTIME) {
+ validateIngestionTransformFunctionVolatility(transformConfig,
existingTransformConfigs);
+ }
Review Comment:
Allowing non-IMMUTABLE transform functions for OFFLINE tables is risky with
the current segment-generation flow because transforms are evaluated during
stats collection and then evaluated again after `RecordReader.rewind()` during
indexing. VOLATILE/STABLE functions (e.g., `now()`, `rand()`) can therefore
yield different values between passes, leading to inconsistent
dictionary/min-max metadata vs indexed values and potentially incorrect
offline-upsert outcomes.
Consider gating non-IMMUTABLE OFFLINE transforms behind an explicit
opt-in/feature flag, or rejecting VOLATILE until segment generation can
materialize/reuse transformed rows (or otherwise guarantee a build-stable
evaluation mode).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
