gianm commented on a change in pull request #10267:
URL: https://github.com/apache/druid/pull/10267#discussion_r469276075
##########
File path:
indexing-service/src/main/java/org/apache/druid/indexing/input/DruidInputSource.java
##########
@@ -87,13 +91,21 @@
@Nullable
private final List<WindowedSegmentId> segmentIds;
private final DimFilter dimFilter;
Review comment:
It's possible to specify a filter alongside transforms today! You can do
it in two places:
- In the `transformSpec` (this works with any input source / format, see
https://druid.apache.org/docs/latest/ingestion/index.html#filter)
- In the druid `inputSource` itself (of course, only works with this input
source)
It's a little silly to have both, perhaps, but there's a practical reason:
specifying a filter in the druid `inputSource` is faster, because it is applied
while creating the cursor that reads the data, and therefore it can use
indexes, etc. The filter in the `transformSpec` is applied _after_ the cursor
generates rows.
But I think in the future, it'd be better to support pushing down the
`transformSpec` filter into the cursor, and then we could deprecate the filter
parameter in the `inputSource`, because it wouldn't be useful anymore.
For now, I suggest we leave it as-is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
