Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24521 )
Change subject: IMPALA-15052: Add read support for unshredded VARIANT values ...................................................................... Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/24521/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24521/3//COMMIT_MSG@17 PS3, Line 17: two We can re-think the FE-side (scalar vs complex), but at the BE-side the current approach has multiple advantages * maps cleanly to how VARIANTs are stored in data files * trivial to implement column readers * trivial to implement VARIANT writers * can re-use STRUCT-related code * zero-copy materialization * zero-copy variant_get() when returning VARIANT * In real worlds scenario a lot of VARIANT value will share the same metadata, the two StringValue approach makes it easy to de-duplicate metadata Implementing late-materialization and codegen doesn't seem to be too hard. Codegen is already there for STRUCT, and VARIANT late-materialization doesn't need full fledged STRUCT late-materialization. It can just delegate SkipRows() to the children. With "an int32 for the length of metadata then copy both metadata and value after it" complicates things too much and requires copying each VARIANT, then it also blocks further optimizations (zero-copy expressions, metadata de-duplication). > shredded vs unshredded > the query uses the whole variant vs just selected fields These are good questions, I have some ideas, but nothing fully carved in stone. But I wouldn't complicate things pre-maturely because of shredded VARIANTs. > the query is non selective/selective with predicates on variant fields / > without predicates on variant fields An adaptive/heuristic copying strategy wouldn't be too hard to implement > the metadata is uniform vs varied Uniform metadata strongly favors 2 StringValue approach. My suspicion is that varied metadata is less common as users probably want to store somewhat similar objects. > it can lead to accumulating multiple decompressed data pages attached to the > buffer What do we do in case of other PLAIN-encoded data? Probably the better approach is to adaptively copy such binary/string data. E.g. when late-materialization is being used then it's probably easy to decide whether to copy or not. > allowing to keep only the used members becomes more important. If we know the > limited number of used members, we don't even need to keep their names, we > can assign a fix numbers to them It's also possible with the current implementation where further optimizations would still apply. But it's more related to future optimizations about variant projection pushdown. We can discuss it more deeply though. -- To view, visit http://gerrit.cloudera.org:8080/24521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2f8a7c9b1d4e5f6a0c3b8d7e9f1a2b4c6d8e0f1 Gerrit-Change-Number: 24521 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Balazs Hevele <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Tue, 07 Jul 2026 16:01:57 +0000 Gerrit-HasComments: Yes
