[
https://issues.apache.org/jira/browse/IMPALA-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17625486#comment-17625486
]
ASF subversion and git services commented on IMPALA-11402:
----------------------------------------------------------
Commit ea6173440c4e3d56ce12c98aca157c4aaa865cbc in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ea6173440 ]
IMPALA-11669: Set TConfiguration during Thrift connection setup
THRIFT-5237 Implement MAX_MESSAGE_SIZE and consolidate limits into a
TConfiguration class. The MAX_MESSAGE_SIZE default to 100MB. This patch
adds backend flag 'thrift_rpc_max_message_size' to override the default
MAX_MESSAGE_SIZE by calling function AssignDefaultTConfiguration.
We set higher default, 1GB, to minimize interruption on existing Impala
workload. We should consider lowering this default once we ensure that
all thrift rpc response can be made in batches, such as explained in
IMPALA-11402.
Thrift tuning for communication with HMS is fixed through HIVE-26633.
Appropriate Hive version bump is required.
Testing:
- Add EXPECT_NO_THROW to verify that 'checkReadBytesAvailable' to does
not throw exception given default FLAGS_thrift_rpc_max_message_size.
- Add MaxMessageSizeFit and MaxMessageSizeExceeded tests in
thrift-server-test.
- Run and pass thrift-server-test
Change-Id: I137683d43c72a34105fd7b32fea3a93532601ae3
Reviewed-on: http://gerrit.cloudera.org:8080/19162
Reviewed-by: Wenzhe Zhou <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> getPartialCatalogObject fails with OOM with huge number of partitions
> ---------------------------------------------------------------------
>
> Key: IMPALA-11402
> URL: https://issues.apache.org/jira/browse/IMPALA-11402
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> The response size of getPartialCatalogObject depends on the number of
> partitions in the request. Even with the optimization of IMPALA-7501, the
> response size could still exceeds the 2GB byte array limit if requesting all
> partitions of a huge table.
> We should add flag to limit the number of partitions in a single
> getPartialCatalogObject request. When more partitions are required, fetch
> them in different batches.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
