clintropolis commented on code in PR #19228:
URL: https://github.com/apache/druid/pull/19228#discussion_r3007353432
##########
processing/src/main/java/org/apache/druid/query/DataSource.java:
##########
@@ -105,6 +106,19 @@ public interface DataSource extends Cacheable
*/
SegmentMapFunction createSegmentMapFunction(Query query);
+ /**
+ * Returns a {@link SegmentPruner} if this datasource embeds in any
information which can be used to determine if a
+ * segment needs processed or not. Note that callers of this method will
always only be processing segments for the
+ * datasource, so there is no need for a 'default' pruner that ensures the
segment has the proper datasource. A return
+ * value of null indicates that no pruning can be performed from this
datasource, though other sources of pruning,
+ * such as filters may still be used.
+ */
+ @Nullable
+ default SegmentPruner createSegmentPruner()
Review Comment:
yea, I thought about this, but ignored it for now since I think
`FilteredDataSource`, and `UnnestDataSource` since it has a filter too, both
need to be a bit more thoughtful in how they prune. I think they need to be
combining with the pruner that is beneath them from the base, but maybe only in
some cases or modifying it in others? Like I think for unnest we might want to
like prune differently if the filter is on the unnest column, depending on
whether unnest is on a mvd or an array, similar to what we do for unnest filter
pushdown? I'm not certain if we have to do anything besides combine
`FilteredDataSource`, I haven't fully thought about it yet, and didn't want to
for now 😅
I'll look into improving this in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
