swaminathanmanish commented on code in PR #14425:
URL: https://github.com/apache/pinot/pull/14425#discussion_r1845494692
##########
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/mapper/SegmentMapper.java:
##########
@@ -71,21 +71,28 @@ public class SegmentMapper {
private final boolean _includeNullFields;
private final int _numSortFields;
private final RecordEnricherPipeline _recordEnricherPipeline;
- private final CompositeTransformer _recordTransformer;
- private final ComplexTypeTransformer _complexTypeTransformer;
+ private final TransformPipeline _transformPipeline;
private final TimeHandler _timeHandler;
private final Partitioner[] _partitioners;
private final String[] _partitionsBuffer;
// NOTE: Use TreeMap so that the order is deterministic
private final Map<String, GenericRowFileManager> _partitionToFileManagerMap
= new TreeMap<>();
- private AdaptiveSizeBasedWriter _adaptiveSizeBasedWriter;
- private List<RecordReaderFileConfig> _recordReaderFileConfigs;
- private List<RecordTransformer> _customRecordTransformers;
+ private final AdaptiveSizeBasedWriter _adaptiveSizeBasedWriter;
+ private final List<RecordReaderFileConfig> _recordReaderFileConfigs;
public SegmentMapper(List<RecordReaderFileConfig> recordReaderFileConfigs,
List<RecordTransformer> customRecordTransformers, SegmentProcessorConfig
processorConfig, File mapperOutputDir) {
+ this(recordReaderFileConfigs,
+ new TransformPipeline(
Review Comment:
Are we clear on how users will pass custom transformers? Up until now our
defaults were covering everything which hid this from users of
SegmentProcessorFramework. If we know for sure what defaults should be
applied, we should pass that here instead of having every SPF user to pass
that.
No-Op/empty list is one usecase but other than that Im not sure how users
will know what exactly to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
