[
https://issues.apache.org/jira/browse/FLINK-41110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib reassigned FLINK-41110:
------------------------------------
Assignee: Ramin Gharib
> Find VARIANT object fields by comparing UTF-8 bytes
> ---------------------------------------------------
>
> Key: FLINK-41110
> URL: https://issues.apache.org/jira/browse/FLINK-41110
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
> Fix For: 2.4.0
>
>
> {{Variant.getField}} decodes every metadata key it compares into a new
> {{String}}:
> {code:java}
> // BinaryVariant.getFieldByKey, linear search for objects below 32 fields
> if (key.equals(getMetadataKey(metadata, id))) {
> {code}
> {{v['a']}} in SQL calls {{getField}} once per record. One lookup in a 13-key
> object can create 13 throwaway strings.
> h3. Proposal
> For objects below {{BINARY_SEARCH_THRESHOLD}} fields, encode the wanted key
> to UTF-8 once. Compare it with the key bytes in the metadata dictionary. The
> builder stores keys as {{key.getBytes(UTF_8)}}, so the lookup encodes the
> same way.
> The binary search for larger objects stays as it is. It depends on the order
> in which the builder sorts keys. That order is {{String.compareTo}}, not the
> UTF-8 byte order the Parquet Variant spec asks for. Fixing the order is a
> separate ticket.
> JMH on JDK 17, 4 lookups per row of 13 keys:
> || ||master||patch||
> |time per row|285 ns|132 ns|
> |allocated per row|1368 B|296 B|
> h3. Behavior changes below the threshold
> ||Lookup||master||patch||
> |{{getField("a\uD800")}} on a field written with that key|null|found|
> |{{getField("\uFFFD")}} on a key whose bytes are not valid UTF-8|found|null|
> Flink never writes keys that are not valid UTF-8. Two distinct malformed keys
> already decoded to the same name.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
