[
https://issues.apache.org/jira/browse/IMPALA-9936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244614#comment-17244614
]
ASF subversion and git services commented on IMPALA-9936:
---------------------------------------------------------
Commit e3e93089d4233785df19a79669bb93b0d23e6d94 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e3e9308 ]
IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog
coordinators
Catalogd RPC response contains the updated catalog objects in a full
form. For instance, a RPC for adding a new partition to an HdfsTable
will return the whole HdfsTable object(metadata) containing all the
partitions. This is required by legacy coordinators where the whole
HdfsTable object is used to replace the stale object(metadata snapshot).
However, LocalCatalog coordinators just need the object names for
invalidations. It's a waste of space to send the full catalog objects to
LocalCatalog coordinators. On the other hand, there is a risk of OOM due
to hitting the Java array limit when serializing a table that has a huge
metadata footprint.
This patch refactors the catalogd RPC responses to only send back
invalidations in need. To distinguish between legacy and LocalCatalog
coordinators, a new field, want_minimal_response, is introduced in
TCatalogServiceRequestHeader which is the header for most of the
Catalogd RPC requests (e.g. TDdlExecRequest, TUpdateCatalogRequest and
TResetMetadataRequest). LocalCatalog coordinators will set this field to
true. When adding updated catalog objects to the response, catalogd will
add invalidations which only contain the object names (e.g. db name,
table name). Note that function objects are small so are ignored in this
optimization.
Tests:
- Add DCHECKs in catalog-op-executor.cc to verify the catalog objects
recieved by LocalCatalog coordinators are in minimal mode.
- Run test_ddl.py in both legacy catalog mode and local catalog mode.
Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Reviewed-on: http://gerrit.cloudera.org:8080/16435
Reviewed-by: Quanlong Huang <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Only send invalidations in DDL responses to LocalCatalog coordinators
> ---------------------------------------------------------------------
>
> Key: IMPALA-9936
> URL: https://issues.apache.org/jira/browse/IMPALA-9936
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
>
> Catalogd RPC requests (TDdlExecRequest, TUpdateCatalogRequest and
> TResetMetadataRequest) should contain the information (whether in
> LocalCatalog mode) of the clients (coordinators). For LocalCatalog
> coordinators, catalogd just need to send back invalidations instead of the
> full catalog objects.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
