[
https://issues.apache.org/jira/browse/IMPALA-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer updated IMPALA-9609:
------------------------------------
Description:
Currently Impala both starts a JVM and creates a Frontend in executor mode
impalads:
JVM:
https://github.com/apache/impala/blob/04fd9ae268d89b07e2a692a916bf2ddcfb2e351b/be/src/service/impalad-main.cc#L61
The 3rd argument of is init_jvm. This is actually needed for some executor
features, e.g. Hive UDFs and HBase.
Frontend:
https://github.com/apache/impala/blob/d877dbc572e95f086b37142e1d41231d5b6b7f9f/be/src/runtime/exec-env.cc#L250
The frontend code itself doesn't really seem to care about whether it serves as
a coordinator.
It would be great to remove the Frontend object from executors completely, but
this may be tricky as Java features used by the executor can have widespread
dependencies.
The critical part is to avoid starting services that are both not needed and
are problematic, e.g. connecting to HMS and Auth services. These can consume
resource and cause errors if not configured well.
The executor only setting is only tested by custom_cluster test, so probably
we'll need to extend those to ensure that HBase/Hive UDFs are not broken.
was:
Currently Impala both starts a JVM and creates a Frontend in executor mode
impalads:
JVM:
https://github.com/apache/impala/blob/04fd9ae268d89b07e2a692a916bf2ddcfb2e351b/be/src/service/impalad-main.cc#L61
The 3rd argument of is init_jvm. This is actually needed for some executor
features, e.g. Hive UDFs and HBase.
Frontend:
https://github.com/apache/impala/blob/d877dbc572e95f086b37142e1d41231d5b6b7f9f/be/src/runtime/exec-env.cc#L250
The frontend code itself doesn't really seem to care about whether it serves as
a coordinator.
It would be great to remove the Frontend object from executors completely, but
this may be tricky as Java features by the executor can have widespread
dependencies.
The critical part is to avoid starting services that are both not needed and
are problematic, e.g. connecting to HMS and Auth services. These can consume
resource and cause errors if not configured well.
The executor only setting is only tested by custom_cluster test, so probably
we'll need to extend those to ensure that HBase/Hive UDFs are not broken.
> Minimize Frontend activity in executor only Impalas
> ---------------------------------------------------
>
> Key: IMPALA-9609
> URL: https://issues.apache.org/jira/browse/IMPALA-9609
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
> Reporter: Csaba Ringhofer
> Priority: Critical
>
> Currently Impala both starts a JVM and creates a Frontend in executor mode
> impalads:
> JVM:
> https://github.com/apache/impala/blob/04fd9ae268d89b07e2a692a916bf2ddcfb2e351b/be/src/service/impalad-main.cc#L61
> The 3rd argument of is init_jvm. This is actually needed for some executor
> features, e.g. Hive UDFs and HBase.
> Frontend:
> https://github.com/apache/impala/blob/d877dbc572e95f086b37142e1d41231d5b6b7f9f/be/src/runtime/exec-env.cc#L250
> The frontend code itself doesn't really seem to care about whether it serves
> as a coordinator.
> It would be great to remove the Frontend object from executors completely,
> but this may be tricky as Java features used by the executor can have
> widespread dependencies.
> The critical part is to avoid starting services that are both not needed and
> are problematic, e.g. connecting to HMS and Auth services. These can consume
> resource and cause errors if not configured well.
> The executor only setting is only tested by custom_cluster test, so probably
> we'll need to extend those to ensure that HBase/Hive UDFs are not broken.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
