npawar commented on a change in pull request #8067:
URL: https://github.com/apache/pinot/pull/8067#discussion_r792178330
##########
File path:
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerFactory.java
##########
@@ -45,30 +50,41 @@ private SegmentPrunerFactory() {
public static List<SegmentPruner> getSegmentPruners(TableConfig tableConfig,
ZkHelixPropertyStore<ZNRecord> propertyStore) {
- RoutingConfig routingConfig = tableConfig.getRoutingConfig();
List<SegmentPruner> segmentPruners = new ArrayList<>();
- // Always prune out empty segments first
- segmentPruners.add(new EmptySegmentPruner(tableConfig, propertyStore));
+ boolean isKinesisEnabled = isKinesisEnabled(tableConfig);
Review comment:
this is leaking immensely specific information into the broker about
Kinesis and the behavior of empty segments. Plus, adding kinesis plugin
dependency in pinot-broker is not the best..
How about one of these options:
1. Add validations to TableConfigUtils.validate, to check that a kinesis
stream table has this pruner added (or if there's any logic in that path which
decorates the table config)
2. Move this method `isKinesisEnabled` to `TableConfigUtils` and rename it
as `needsEmptySegmentPruner`. Part of that, check if routingTypes already has
EmptySegmentPruner, if not check if kinesis. Possibly even add
"needsEmptySegmetPruner" to StreamConfig
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
