npawar commented on a change in pull request #8067:
URL: https://github.com/apache/pinot/pull/8067#discussion_r792174926
##########
File path: pinot-broker/pom.xml
##########
@@ -145,5 +145,11 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.pinot</groupId>
+ <artifactId>pinot-kinesis</artifactId>
+ <version>0.10.0-SNAPSHOT</version>
+ <scope>compile</scope>
Review comment:
remove hardcoded version
##########
File path:
pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerTest.java
##########
@@ -188,36 +178,31 @@ public void testSegmentPrunerFactoryForTimeRangePruner() {
// Routing config is missing
List<SegmentPruner> segmentPruners =
SegmentPrunerFactory.getSegmentPruners(tableConfig, _propertyStore);
- assertEquals(segmentPruners.size(), 1);
- assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
+ assertEquals(segmentPruners.size(), 0);
// Segment pruner type is not configured
RoutingConfig routingConfig = mock(RoutingConfig.class);
when(tableConfig.getRoutingConfig()).thenReturn(routingConfig);
segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig,
_propertyStore);
- assertEquals(segmentPruners.size(), 1);
- assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
+ assertEquals(segmentPruners.size(), 0);
// Validation config is missing
when(routingConfig.getSegmentPrunerTypes()).thenReturn(
Collections.singletonList(RoutingConfig.TIME_SEGMENT_PRUNER_TYPE));
segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig,
_propertyStore);
- assertEquals(segmentPruners.size(), 1);
- assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
+ assertEquals(segmentPruners.size(), 0);
// Time column is missing
SegmentsValidationAndRetentionConfig validationConfig =
mock(SegmentsValidationAndRetentionConfig.class);
when(tableConfig.getValidationConfig()).thenReturn(validationConfig);
segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig,
_propertyStore);
- assertEquals(segmentPruners.size(), 1);
- assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
+ assertEquals(segmentPruners.size(), 0);
// Time range pruner should be returned
when(validationConfig.getTimeColumnName()).thenReturn(TIME_COLUMN);
segmentPruners = SegmentPrunerFactory.getSegmentPruners(tableConfig,
_propertyStore);
- assertEquals(segmentPruners.size(), 2);
- assertTrue(segmentPruners.get(0) instanceof EmptySegmentPruner);
- assertTrue(segmentPruners.get(1) instanceof TimeSegmentPruner);
+ assertEquals(segmentPruners.size(), 1);
+ assertTrue(segmentPruners.get(0) instanceof TimeSegmentPruner);
}
Review comment:
some test to check that EmptySegmentPruner is added as expected if 1.
already in table config 2. because streaming table with isEmptySegment?
##########
File path:
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerFactory.java
##########
@@ -45,30 +50,41 @@ private SegmentPrunerFactory() {
public static List<SegmentPruner> getSegmentPruners(TableConfig tableConfig,
ZkHelixPropertyStore<ZNRecord> propertyStore) {
- RoutingConfig routingConfig = tableConfig.getRoutingConfig();
List<SegmentPruner> segmentPruners = new ArrayList<>();
- // Always prune out empty segments first
- segmentPruners.add(new EmptySegmentPruner(tableConfig, propertyStore));
+ boolean isKinesisEnabled = isKinesisEnabled(tableConfig);
Review comment:
this is leaking immensely specific information into the broker about
Kinesis and the behavior of empty segments. Plus, adding kinesis plugin
dependency in pinot-broker is not the best..
How about one of these options:
1. Add validations to TableConfigUtils.validate, to check that a kinesis
stream table has this pruner added (or if there's any logic in that path which
decorates the table config)
2. Move this mthod `isKinesisEnabled` to `TableConfigUtils` ad rename it as
`needsEmptySegmentPruner`. Part of that, check if routingTypes already has
EmptySegmentPruner, if not check if kinesis. Possibly even add
"needsEmptySegmetPruner" to StreamConfig
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
