[
https://issues.apache.org/jira/browse/HIVE-28026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghav Aggarwal updated HIVE-28026:
-----------------------------------
Description:
{*}Query{*}: select * from _<table_name>_
{*}Explanation{*}:
On running the above mentioned query on a hive proto table, multiple tez
containers will be spawned to process the data. In a container, if there are
multiple hdfs splits and the combined size of decompressed data is more than
2GB then the query fails with the following error:
{code:java}
"While parsing a protocol message, the input ended unexpectedly in the middle
of a field. This could mean either that the input has been truncated or that
an embedded message misreported its own length." {code}
This is happening because of
_[CodedInputStream|https://github.com/protocolbuffers/protobuf/blob/54489e95e01882407f356f83c9074415e561db00/java/core/src/main/java/com/google/protobuf/CodedInputStream.java#L2712C7-L2712C16]
i.e. byteLimit += totalBytesRetired + pos;_
_byteLimit_ is __ getting InterOverflow as _totalBytesRetired_ is retaining
count of all the bytes that it has read as CodedInputStream is initiliazed once
for a container
[https://github.com/apache/hive/blob/564d7e54d2360488611da39d0e5f027a2d574fc1/ql/src/java/org/apache/tez/dag/history/logging/proto/ProtoMessageWritable.java#L96]
.
This is different from issue reproduced in
[https://github.com/zabetak/protobuf-large-message] as there it is a single
proto data file more than 2GB, but in my case, there are multiple file total
resulting in 2GB.
CC [~zabetak]
*Limitation:*
This fix will still not resolve the issue which is mentioned
[https://github.com/protocolbuffers/protobuf/issues/11729]
Here is table description:
{code:java}
+-------------------------------+----------------------------------------------------+----------------------------------------------------+
| col_name | data_type
| comment |
+-------------------------------+----------------------------------------------------+----------------------------------------------------+
| # col_name | data_type
| comment |
| eventtype | string
| from deserializer |
| hivequeryid | string
| from deserializer |
| timestamp | bigint
| from deserializer |
| executionmode | string
| from deserializer |
| requestuser | string
| from deserializer |
| queue | string
| from deserializer |
| user | string
| from deserializer |
| operationid | string
| from deserializer |
| tableswritten | array<string>
| from deserializer |
| tablesread | array<string>
| from deserializer |
| otherinfo | array<struct<key:string,value:string>>
| from deserializer |
| | NULL
| NULL |
| # Partition Information | NULL
| NULL |
| # col_name | data_type
| comment |
| date | string
| |
| | NULL
| NULL |
| # Detailed Table Information | NULL
| NULL |
| Database: | raghav
| NULL |
| OwnerType: | USER
| NULL |
| Owner: | hive
| NULL |
| CreateTime: | Tue Dec 19 11:05:48 PST 2023
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Retention: | 0
| NULL |
| Location: | hdfs://hostname:8020/user/raghav/query_data
| NULL |
| Table Type: | EXTERNAL_TABLE
| NULL |
| Table Parameters: | NULL
| NULL |
| | EXTERNAL
| TRUE |
| | bucketing_version
| 2 |
| | discover.partitions
| true |
| | numFiles
| 14 |
| | numPartitions
| 7 |
| | numRows
| 0 |
| | proto.class
| org.apache.hadoop.hive.ql.hooks.proto.HiveHookEvents$HiveHookEventProto |
| | proto.maptypes
| org.apache.hadoop.hive.ql.hooks.proto.MapFieldEntry |
| | rawDataSize
| 0 |
| | totalSize
| 62336619013 |
| | transient_lastDdlTime
| 1703012748 |
| | NULL
| NULL |
| # Storage Information | NULL
| NULL |
| SerDe Library: |
org.apache.hadoop.hive.ql.io.protobuf.ProtobufMessageSerDe | NULL
|
| InputFormat: |
org.apache.hadoop.hive.ql.io.protobuf.ProtobufMessageInputFormat | NULL
|
| OutputFormat: |
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
|
| Compressed: | No
| NULL |
| Num Buckets: | -1
| NULL |
| Bucket Columns: | []
| NULL |
| Sort Columns: | []
| NULL |
| Storage Desc Params: | NULL
| NULL |
| | serialization.format
| 1 |
+-------------------------------+----------------------------------------------------+----------------------------------------------------+
{code}
was:
{*}Query{*}: select * from _<table_name>_
{*}Explanation{*}:
On running the above mentioned query on a hive proto table, multiple tez
containers will be spawned to process the data. In a container, if there are
multiple hdfs splits and the combined size of decompressed data is more than
2GB then the query fails with the following error:
{code:java}
"While parsing a protocol message, the input ended unexpectedly in the middle
of a field. This could mean either that the input has been truncated or that
an embedded message misreported its own length." {code}
This is happening because of
_[CodedInputStream|https://github.com/protocolbuffers/protobuf/blob/54489e95e01882407f356f83c9074415e561db00/java/core/src/main/java/com/google/protobuf/CodedInputStream.java#L2712C7-L2712C16]
i.e. byteLimit += totalBytesRetired + pos;_
_byteLimit_ is __ getting InterOverflow as _totalBytesRetired_ is retaining
count of all the bytes that it has read as CodedInputStream is initiliazed once
for a container
[https://github.com/apache/hive/blob/564d7e54d2360488611da39d0e5f027a2d574fc1/ql/src/java/org/apache/tez/dag/history/logging/proto/ProtoMessageWritable.java#L96]
.
This is different from issue reproduced in
[https://github.com/zabetak/protobuf-large-message] as there it is a single
proto data file more than 2GB, but in my case, there are multiple file total
resulting in 2GB.
CC [~zabetak]
*Limitation:*
This fix will still not resolve the issue which is mentioned
[https://github.com/protocolbuffers/protobuf/issues/11729]
> Reading proto data more than 2GB from multiple splits fails
> -----------------------------------------------------------
>
> Key: HIVE-28026
> URL: https://issues.apache.org/jira/browse/HIVE-28026
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.0.0-beta-1
> Environment:
> Reporter: Raghav Aggarwal
> Assignee: Raghav Aggarwal
> Priority: Major
> Labels: pull-request-available
>
> {*}Query{*}: select * from _<table_name>_
> {*}Explanation{*}:
> On running the above mentioned query on a hive proto table, multiple tez
> containers will be spawned to process the data. In a container, if there are
> multiple hdfs splits and the combined size of decompressed data is more than
> 2GB then the query fails with the following error:
> {code:java}
> "While parsing a protocol message, the input ended unexpectedly in the middle
> of a field. This could mean either that the input has been truncated or that
> an embedded message misreported its own length." {code}
>
> This is happening because of
> _[CodedInputStream|https://github.com/protocolbuffers/protobuf/blob/54489e95e01882407f356f83c9074415e561db00/java/core/src/main/java/com/google/protobuf/CodedInputStream.java#L2712C7-L2712C16]
> i.e. byteLimit += totalBytesRetired + pos;_
> _byteLimit_ is __ getting InterOverflow as _totalBytesRetired_ is retaining
> count of all the bytes that it has read as CodedInputStream is initiliazed
> once for a container
> [https://github.com/apache/hive/blob/564d7e54d2360488611da39d0e5f027a2d574fc1/ql/src/java/org/apache/tez/dag/history/logging/proto/ProtoMessageWritable.java#L96]
> .
>
> This is different from issue reproduced in
> [https://github.com/zabetak/protobuf-large-message] as there it is a single
> proto data file more than 2GB, but in my case, there are multiple file total
> resulting in 2GB.
> CC [~zabetak]
> *Limitation:*
> This fix will still not resolve the issue which is mentioned
> [https://github.com/protocolbuffers/protobuf/issues/11729]
> Here is table description:
>
> {code:java}
> +-------------------------------+----------------------------------------------------+----------------------------------------------------+
> | col_name | data_type
> | comment |
> +-------------------------------+----------------------------------------------------+----------------------------------------------------+
> | # col_name | data_type
> | comment |
> | eventtype | string
> | from deserializer |
> | hivequeryid | string
> | from deserializer |
> | timestamp | bigint
> | from deserializer |
> | executionmode | string
> | from deserializer |
> | requestuser | string
> | from deserializer |
> | queue | string
> | from deserializer |
> | user | string
> | from deserializer |
> | operationid | string
> | from deserializer |
> | tableswritten | array<string>
> | from deserializer |
> | tablesread | array<string>
> | from deserializer |
> | otherinfo | array<struct<key:string,value:string>>
> | from deserializer |
> | | NULL
> | NULL |
> | # Partition Information | NULL
> | NULL |
> | # col_name | data_type
> | comment |
> | date | string
> | |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | raghav
> | NULL |
> | OwnerType: | USER
> | NULL |
> | Owner: | hive
> | NULL |
> | CreateTime: | Tue Dec 19 11:05:48 PST 2023
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Retention: | 0
> | NULL |
> | Location: | hdfs://hostname:8020/user/raghav/query_data
> | NULL |
> | Table Type: | EXTERNAL_TABLE
> | NULL |
> | Table Parameters: | NULL
> | NULL |
> | | EXTERNAL
> | TRUE |
> | | bucketing_version
> | 2 |
> | | discover.partitions
> | true |
> | | numFiles
> | 14 |
> | | numPartitions
> | 7 |
> | | numRows
> | 0 |
> | | proto.class
> |
> org.apache.hadoop.hive.ql.hooks.proto.HiveHookEvents$HiveHookEventProto |
> | | proto.maptypes
> | org.apache.hadoop.hive.ql.hooks.proto.MapFieldEntry |
> | | rawDataSize
> | 0 |
> | | totalSize
> | 62336619013 |
> | | transient_lastDdlTime
> | 1703012748 |
> | | NULL
> | NULL |
> | # Storage Information | NULL
> | NULL |
> | SerDe Library: |
> org.apache.hadoop.hive.ql.io.protobuf.ProtobufMessageSerDe | NULL
> |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.protobuf.ProtobufMessageInputFormat | NULL
> |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
> |
> | Compressed: | No
> | NULL |
> | Num Buckets: | -1
> | NULL |
> | Bucket Columns: | []
> | NULL |
> | Sort Columns: | []
> | NULL |
> | Storage Desc Params: | NULL
> | NULL |
> | | serialization.format
> | 1 |
> +-------------------------------+----------------------------------------------------+----------------------------------------------------+
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
