[
https://issues.apache.org/jira/browse/IMPALA-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091960#comment-17091960
]
ASF subversion and git services commented on IMPALA-9609:
---------------------------------------------------------
Commit 58273fff601dcc763ac43f7cc275a174a2e18b6b in impala's branch
refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=58273ff ]
IMPALA-9609: Minimize Frontend activity in executor only impalads
Until now the Frontend started fully regardless of flag is_coordinator,
e.g. created connections to the HMS, which is both error prone and can
DoS the metastore. (note that even coordinators started to connect to
HMS only in the recent past, related to local catalog mode and ACID
transactions)
Executor only impalads still need a JVM as queries can contain
java calls (HDFS/Hbase API calls, Hive UDFs), but most of the JNI API
provided by JniFrontend shouldn't be called by executors. It seems
that the whole Frontend object is needed only by coordinators.
Testing:
- generally executor only mode doesn't seem to be well covered
- ran test_coordinators.py which has some tests with executor only
impalads
- added new test for HBase tables (Hive UDFs and HDFS were already
covered)
Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1
Reviewed-on: http://gerrit.cloudera.org:8080/15793
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Minimize Frontend activity in executor only Impalas
> ---------------------------------------------------
>
> Key: IMPALA-9609
> URL: https://issues.apache.org/jira/browse/IMPALA-9609
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
> Reporter: Csaba Ringhofer
> Assignee: Csaba Ringhofer
> Priority: Critical
>
> Currently Impala both starts a JVM and creates a Frontend in executor mode
> impalads:
> JVM:
> https://github.com/apache/impala/blob/04fd9ae268d89b07e2a692a916bf2ddcfb2e351b/be/src/service/impalad-main.cc#L61
> The 3rd argument of is init_jvm. This is actually needed for some executor
> features, e.g. Hive UDFs and HBase.
> Frontend:
> https://github.com/apache/impala/blob/d877dbc572e95f086b37142e1d41231d5b6b7f9f/be/src/runtime/exec-env.cc#L250
> The frontend code itself doesn't really seem to care about whether it serves
> as a coordinator.
> It would be great to remove the Frontend object from executors completely,
> but this may be tricky as Java features used by the executor can have
> widespread dependencies.
> The critical part is to avoid starting services that are both not needed and
> are problematic, e.g. connecting to HMS and Auth services. These can consume
> resource and cause errors if not configured well.
> The executor only setting is only tested by custom_cluster test, so probably
> we'll need to extend those to ensure that HBase/Hive UDFs are not broken.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
