barbarj opened a new pull request, #24382: URL: https://github.com/apache/datafusion/pull/24382
## Which issue does this PR close? - Closes #24277 ## Rationale for this change In order for DataFusion Distributed (and presumably other distributed DF projects) to make use of `HashTableLookupExpr` as a dynamic filter across network boundaries, it needs to be serializable. ## What changes are included in this PR? The only sticky part of serializing `HashTableLookupExpr` is its `Map`. The inner members of the two variants of `Map`, `HashMap` and `ArrayMap` both support much more functionality than is needed to evaluate this as a dynamic expression. So, in order to simply serialization and minimize the on-wire size, we serialize only the aspects required for expression evaluation (i.e. membership checks) We replace `HashTableLookupExpr`'s `Map` with a local version that includes the membership-only variants. These are implemented such that they are only constructible via deserialization. ## Are these changes tested? The `roundtrip_hash_table_lookup_expr_to_lit` test is replaced with two regular roundtrip tests, one each for the `HashMap` and `ArrayMap` versions of `HashTableLookupExpr`. This PR also adds a bunch of tests testing the post-deserialization behavior of the new membership-only variants. ## Are there any user-facing changes? - `JoinHashMapType` has a new required method `hashes` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
