snleee commented on a change in pull request #6094:
URL: https://github.com/apache/incubator-pinot/pull/6094#discussion_r501260008
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/framework/SegmentConfig.java
##########
@@ -31,13 +31,28 @@
public class SegmentConfig {
private static final int DEFAULT_MAX_NUM_RECORDS_PER_SEGMENT = 5_000_000;
+ private static final String DEFAULT_SEGMENT_NAME_GENERATOR_TYPE = "simple";
+
private final int _maxNumRecordsPerSegment;
- // TODO: more configs such as segment name prefix
+
+ // Currently, 'simple', 'normalizedDate' are supported
+ private final String _segmentNameGeneratorType;
+ private final String _segmentPrefix;
+ private final String _segmentPostfix;
+ private final boolean _excludeSequenceId;
@JsonCreator
- private SegmentConfig(@JsonProperty(value = "maxNumRecordsPerSegment") int
maxNumRecordsPerSegment) {
+ private SegmentConfig(@JsonProperty(value = "maxNumRecordsPerSegment") int
maxNumRecordsPerSegment,
Review comment:
SegmentConfig already includes `@JsonIgnoreProperties(ignoreUnknown =
true)` at the top. Also, this segment processing framework is checked in very
recently and it's still under development. Until we put this on the production,
we can make the changes and not need to worry about backward compatibility yet.
I'm working closely with @npawar on improving segment processor framework. This
will be used for both `RealtimeToOfflineTask` & `SegmentMergeRollup`
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/framework/SegmentConfig.java
##########
@@ -31,13 +31,28 @@
public class SegmentConfig {
private static final int DEFAULT_MAX_NUM_RECORDS_PER_SEGMENT = 5_000_000;
+ private static final String DEFAULT_SEGMENT_NAME_GENERATOR_TYPE = "simple";
+
private final int _maxNumRecordsPerSegment;
- // TODO: more configs such as segment name prefix
+
+ // Currently, 'simple', 'normalizedDate' are supported
+ private final String _segmentNameGeneratorType;
+ private final String _segmentPrefix;
+ private final String _segmentPostfix;
+ private final boolean _excludeSequenceId;
@JsonCreator
- private SegmentConfig(@JsonProperty(value = "maxNumRecordsPerSegment") int
maxNumRecordsPerSegment) {
+ private SegmentConfig(@JsonProperty(value = "maxNumRecordsPerSegment") int
maxNumRecordsPerSegment,
Review comment:
SegmentConfig already includes `@JsonIgnoreProperties(ignoreUnknown =
true)` at the top. Also, this segment processing framework is checked in very
recently and it's still under development. Until we put this on the production,
we can make the changes and not need to worry about the backward compatibility
yet. I'm working closely with @npawar on improving segment processor framework.
This will be used for both `RealtimeToOfflineTask` & `SegmentMergeRollup`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
