RussellSpitzer commented on issue #3607: URL: https://github.com/apache/iceberg/issues/3607#issuecomment-1024416788
> Hi @RussellSpitzer , sorry to jump on this thread, but I have a question regarding your last message, if you could elaborate, as I think I'm missing a step: > > > For example if you say `purchase_ts = timestampOf(2021-01-01)` and you have actually partitioning on `day(purchase_ts)` it transforms the predicate into `day(purchase_ts) = day(timestampOf(2021-01-01)`. > > Surely, `purchase_ts` being a timestamp, has all sorts of values that are not a round day, so to have a predicate transformed into `day(purchase_ts) = day(timestampOf(2021-01-01)`, how would you write it? > > * `on target.purchase_ts = timestampOf(2021-01-01)` doesn't seem right, because if `purchase_ts = 2021-01-01 14:10:00.001`, the meaning is actually different > * `on target.purchase_ts = timestamp '2021-01-01 14:10:00.001'`, if you know the exact value of the partition key in the target row > * `on target.purchase_ts >= timestamp '2021-01-01' and target.purchase_ts < timestamp '2021-01-02'` > * I don't think `on day(target.purchase_ts) = '2021-01-01'` or similar works, as far as I've tried? > > Thanks! I think I may have misled you by oversimplifying. The user here still only writes queries using their exact restrictions, Iceberg then uses this restriction to create restrictions which match the partitioning. For example, Iceberg knows a specific timestamp can only occur in a certain day and it can use that information to limit the files read. Iceberg doesn't disregard the original predicate, that stays with the execution engine for actually evaluating rows but Iceberg can still use this timestamp for partition pruning and file evaluation. For example say you are looking for `ts = 3PM on Aug 12` First thing we do is look at our manifest_list file, see docs in the [spec](https://iceberg.apache.org/#spec/) Each entry there will have a `partitions` field summary column and a `partition_spec_id` to let us know how to use that data. We load up the partition spec for the given spec ID and transform the original predicate into one that matches that spec. If our `spec` contains a `day(ts)` transform we take the original predicate and transform it using the `day` transform . So for evaluating this line the original predicate becomes `day(ts) = projectDay(3pm Aug 12) = Aug 12`. With this new transform we evaluate all the `partitions` listed in this file. These values only contain `day(ts)` since that is the only thing kept by the spec. If any pass we know the particular manifest file may have valid datafiles to be scanned. Once we have a list of all the possible manifest files that may have hits we play this game again. Now we check against `ManifestEntries`. Every entry contains a `partition` value and then details about the `datafile` (spec_id is inherited from the entry in manifest_list). Here we can do two steps of evaluation for each individual data file. First we can use the transformed predicate (`day(ts) = Aug 12`) to check if the partition value is a match, if so we then move to evaluating the individual metrics of the file. Here we would use the original predicate and would check whether the `3PM on Aug 12` is a possible value for the timestamp column of each file based on the min and max values for that column. If datafile passes both of these checks we keep it for the scan. The scan then contains of all data files which we know **may** have our given row, this is transformed into a set of tasks for whatever execution engine is in use and evaluated. The execution engine then will use its own logic to filter individual rows with the original predicates (YMMV based on engine specific implementations). So what happens if we cannot transform a predicate into the partition spec? Or what if a data file was inserted into the table when it was unpartitioned? In both of these cases we default to "this file may contain the row we are looking for" and return it to the engine. For example, suppose you have a predicate `age > 10` and a partition spec of `bucket(age,128)`. There is no way to project a [greater than predicate](https://github.com/apache/iceberg/blob/f960698a04266270a8eaf9e23e8b55e97e550564/api/src/main/java/org/apache/iceberg/transforms/Bucket.java#L131-L134) into a valid bucket predicate so we simply have to say all partitions may match. ### My big TLDR here is: As a user you query on normal columns, Iceberg attempts to transform your predicates into ones that match the partitioning of the files within the table to prune out files. When Iceberg cannot transform the predicates it simply assumes there may be a match and returns those files to the execution engine which does the actual row level filtering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
