[
https://issues.apache.org/jira/browse/NIFI-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Wicks updated NIFI-6175:
------------------------------
Description:
The Livy Session Controller is missing many of the options available, and many
of them I feel are critical for this service to be useful (queue? conf? num of
executors?)
* Would like to see all available options there, with a blanket "conf" option
for users to provide custom configuration.
* When the controller service shuts down, sessions are left running, with no
option to shut them down. Add in functionality to shutdown open sessions.
* If the controller service finds no Idle Livy Sessions, it will create a new
session... until the queue runs out of resources :). Need to have a
Min/Max/Should be elastic or strict option
* When Livy starts up, it searches for existing sessions, but does not verify
that those sessions belong to it.
** The Kerberos identity should be used to verify the identity on the session
matches the identity on the controller service.
** Also, if a Proxy user has been specified, that should also be verified. If
no proxy user was specified, then the Proxy user on the Livy session should
match the Kerberos identity.
* The initialization of the SSL Context is not implemented in a thread safe
way. This leads to exceptions when multiple threads are running against the
same Controller Service.
** SSL Context init should be made thread safe.
Livy should also support Batch mode.
* Include a controller service to re-use configs, but controller service is
basically just a config holder
* Processor named `ExecuteSparkBatch`. This is harder than Session because
Batch mode only supports code submission through a file path. So users will
need to upload to HDFS first.
was:
The Livy Session Controller is missing many of the options available, and many
of them I feel are critical for this service to be useful (queue? conf? num of
executors?)
* Would like to see all available options there, with a blanket "conf" option
for users to provide custom configuration.
* When the controller service shuts down, sessions are left running, with no
option to shut them down. Add in functionality to shutdown open sessions.
* If the controller service finds no Idle Livy Sessions, it will create a new
session... until the queue runs out of resources :). Need to have a
Min/Max/Should be elastic or strict option
* When Livy starts up, it searches for existing sessions, but does not verify
that those sessions belong to it.
** The Kerberos identity should be used to verify the identity on the session
matches the identity on the controller service.
** Also, if a Proxy user has been specified, that should also be verified. If
no proxy user was specified, then the Proxy user on the Livy session should
match the Kerberos identity.
* The initialization of the SSL Context is not implemented in a thread safe
way. This leads to exceptions when multiple threads are running against the
same Controller Service.
** SSL Context init should be made thread safe.
> Spark Livy - Add Support for All Missing Session Features
> ---------------------------------------------------------
>
> Key: NIFI-6175
> URL: https://issues.apache.org/jira/browse/NIFI-6175
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.9.2
> Reporter: Peter Wicks
> Assignee: Peter Wicks
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Livy Session Controller is missing many of the options available, and
> many of them I feel are critical for this service to be useful (queue? conf?
> num of executors?)
> * Would like to see all available options there, with a blanket "conf"
> option for users to provide custom configuration.
> * When the controller service shuts down, sessions are left running, with no
> option to shut them down. Add in functionality to shutdown open sessions.
> * If the controller service finds no Idle Livy Sessions, it will create a
> new session... until the queue runs out of resources :). Need to have a
> Min/Max/Should be elastic or strict option
> * When Livy starts up, it searches for existing sessions, but does not
> verify that those sessions belong to it.
> ** The Kerberos identity should be used to verify the identity on the
> session matches the identity on the controller service.
> ** Also, if a Proxy user has been specified, that should also be verified.
> If no proxy user was specified, then the Proxy user on the Livy session
> should match the Kerberos identity.
> * The initialization of the SSL Context is not implemented in a thread safe
> way. This leads to exceptions when multiple threads are running against the
> same Controller Service.
> ** SSL Context init should be made thread safe.
>
> Livy should also support Batch mode.
> * Include a controller service to re-use configs, but controller service is
> basically just a config holder
> * Processor named `ExecuteSparkBatch`. This is harder than Session because
> Batch mode only supports code submission through a file path. So users will
> need to upload to HDFS first.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
