AkshayUpadhyay2411 opened a new issue, #19819:
URL: https://github.com/apache/pinot/issues/19819
`SegmentGenerationAndPushTask` can append a UUID to generated segment names
(introduced in #10335) so repeated minion task runs do not collide on segment
names. Tables that use deterministic segment name generators (`inputFile`,
`simple`, `normalizedDate`) may set `append.uuid.to.segment.name` to `false` so
each scheduled run refreshes the same segment name instead of creating a new
UUID-suffixed segment every time.
`BatchConfig` reads the setting from the segment name generator props
(`segmentNameGenerator.configs.append.uuid.to.segment.name`), and the task also
accepts a top-level `append.uuid.to.segment.name` key. In
`SegmentGenerationAndPushTask`, neither can currently be set to `false`,
because two places override the configured value:
### 1. Task generator (`SegmentGenerationAndPushTaskGenerator`)
`getSingleFileGenerationTaskConfig()` unconditionally set the top-level task
key `append.uuid.to.segment.name = true`, ignoring an explicit
**`segmentNameGenerator.configs.append.uuid.to.segment.name`** in the batch map.
### 2. Task executor (`SegmentGenerationAndPushTaskExecutor`)
`generateTaskSpec()` loaded prefixed generator configs, then unconditionally
`addConfig(...)` from the top-level task key, overwriting prefixed values.
### Expected behavior
- An explicit `append.uuid.to.segment.name` (top-level) or
`segmentNameGenerator.configs.append.uuid.to.segment.name`
in `batchConfigMaps` is respected; the prefixed key wins if both are set.
- If neither is set, the default stays `true` (collision avoidance from
#10335).
### Context
During review of #10335, snleee asked whether `APPEND_UUID_TO_SEGMENT_NAME`
was already supported on the segment name
generator; the author confirmed it worked for the basic case, but explicitly
pre-set values were not covered.
## Proposed fix
- Generator: default to `true` only when neither key is set; carry the
prefixed value into the top-level task key.
- Executor: only fall back to the top-level key when the prefixed config is
absent.
Unit tests for each case plus a generator -> executor round trip. No config
keys are added or removed.
Related: #16909 (open) touches the same generator line with a different
approach (top-level key, default `false`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
