[
https://issues.apache.org/jira/browse/IMPALA-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866883#comment-17866883
]
ASF subversion and git services commented on IMPALA-13194:
----------------------------------------------------------
Commit daa4d6e916f80d8b929dcf4873668accceb33b0b in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=daa4d6e91 ]
IMPALA-13194: Fast-serialize position delete records
Currently the serialization of position delete records are very
wasteful. The records contain slots 'file_path' and 'pos'. And
what we do during serialization is the following:
1. Write fixed-size tuples that have a StringValue and a BigInt slot
2. Copy the StringValue's contents after the tuple.
3. Convert the StringValue ptr to be an offset to the string data
So we end up having something like this:
+-------------+--------+----------------+-------------+--------+----------------+-----+
| StringValue | BigInt | File path | StringValue | BigInt | File path
| ... |
+-------------+--------+----------------+-------------+--------+----------------+-----+
| ptr, len | 42 | /.../a.parquet | ptr, len | 43 | /.../a.parquet
| ... |
+-------------+--------+----------------+-------------+--------+----------------+-----+
This is very redundant to store the file paths that way, and in the
end we will have a huge buffer that we need to compress and send over
the network. Moreover, we copy the file paths in memory twice:
1. From input row batch to the KrpcDataStreamSender::Channel's temporary row
batch
2. From the temporary row batch to the outbound row batch (during
serialization)
The position delete files store the delete records in ascending order.
This means adjacent records mostly have the same file path. So we could
just buffer the position delete records up to the Channel's capacity,
then serialize the data in a more efficient way.
With this patch, serialized data will look like this:
+----------------+-------------+--------+-------------+--------+-----+
| File path | StringValue | BigInt | StringValue | BigInt | ... |
+----------------+-------------+--------+-------------+--------+-----+
| /.../a.parquet | ptr, len | 42 | ptr, len | 43 | ... |
+----------------+-------------+--------+-------------+--------+-----+
File path, then tuples with the same file path, after that comes the
next file path and tuples associated with that one, and so on.
Measurements:
07:EXCHANGE : 1m ==> 52s
F02:EXCHANGE SENDER: 1m2s ==> 16s
Change-Id: I6095f318e3d06dedb4197681156b40dd2a326c6f
Reviewed-on: http://gerrit.cloudera.org:8080/21563
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Fast-serialize position delete records
> --------------------------------------
>
> Key: IMPALA-13194
> URL: https://issues.apache.org/jira/browse/IMPALA-13194
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Currently the serialization of position delete records are very wasteful. The
> records contain slots 'file_path' and 'pos'. And what we do during
> serialization is the following.
> # Write fixed-size tuple that have a StringValue and a BigInt slot (20 bytes
> in total)
> # We copy the StringValue's contents after the tuple.
> # We convert the StringValue slot to be an offset to the string data
> So we end up having something like this:
> {noformat}
> +-------------+--------+----------------+-------------+--------+----------------+-----+
> | StringValue | BigInt | File path | StringValue | BigInt | File path
> | ... |
> +-------------+--------+----------------+-------------+--------+----------------+-----+
> | ptr, len | 42 | /.../a.parquet | ptr, len | 43 |
> /.../a.parquet | ... |
> +-------------+--------+----------------+-------------+--------+----------------+-----+
> {noformat}
> This is very redundant to store the file paths that way, and at the end we
> will have a huge buffer that we need to compress and send over the network.
> Moreover, we copy the file paths in memory twice:
> # From input row batch to the KrpcDataStreamSender::Channel's temporary row
> batch
> # From the temporary row batch to the outbound row batch (during
> serialization)
> The position delete files store the delete records in ascending order. This
> means adjacent records mostly have the same file path. So we could just
> buffer the position delete records up to the Channel's capacity, then
> serialize the data in a more efficient way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
