maytasm commented on PR #18531: URL: https://github.com/apache/druid/pull/18531#issuecomment-3299977331
> > How is this different than setting `intervals` in `granularitySpec`? > > @maytasm Thanks for taking a look! This filter is at the input source level which means it controls the data file paths pulled from the iceberg table for ingestion. Pruning the data files during the iceberg table scan phase helps reduce scan times and also reduces the number of files processed by the subsequent indexing phase. I see. Not directly related to this PR (time window filter) but Iceberg filter in general, what if the filterColumn is not a partition column? This would result in files that may contain rows not matching the filter right? We would then need to filter it on Druid side again using filter in transformSpec? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
