Nilesh Rathi created LIVY-1061:
----------------------------------
Summary: Add livy.server.app-name-prefix configuration for
interactive session YARN application names
Key: LIVY-1061
URL: https://issues.apache.org/jira/browse/LIVY-1061
Project: Livy
Issue Type: Improvement
Reporter: Nilesh Rathi
Currently, Apache Livy assigns a hardcoded YARN application name to interactive
Spark sessions when no user-provided name is supplied in the session creation
request. The default name follows the format {{livy-session-<id>}}, and there
is no server-level configuration to override or customize this prefix.
This creates a limitation for cluster administrators, especially in High
Availability (HA) or multi-cluster environments, where Livy sessions launched
across different Livy server instances or clusters all appear with identical
naming patterns in the YARN Resource Manager UI. Operations and monitoring
teams need a way to distinguish Livy-managed applications at the cluster or
environment level without requiring every client to pass a custom name in each
REST API request.
h3. Current Behavior
* *Interactive sessions:* When a client creates a session via {{POST
/sessions}} without specifying a {{name}} field, Livy sets {{spark.app.name}}
to {{livy-session-<id>}} (in {{InteractiveSession.scala}}).
* *Batch sessions:* Users can already set the application name via the {{name}}
field in {{POST /batches}} requests.
h3. Expected Behavior
Add a new Livy server configuration property, {{livy.server.app-name-prefix}},
that allows administrators to define a custom prefix for YARN application names
of interactive sessions.
* When unset (default): retain existing behavior — application names remain
{{livy-session-<id>}}.
* When set (e.g., {{livy.server.app-name-prefix=my-cluster}}): interactive
session YARN application names use the configured prefix — {{my-cluster-<id>}}.
h3. Scope
* *Interactive sessions only.* Batch sessions already support per-request
naming via the REST API {{name}} field; changing batch behavior could break
existing automation or monitoring that relies on current naming conventions.
h3. Use Cases
* HA deployments with multiple Livy server instances where administrators need
to identify which cluster or environment a session belongs to in the YARN RM UI.
* Shared YARN clusters where ops teams want a consistent, cluster-specific
prefix for all Livy interactive workloads.
* Monitoring and alerting systems that filter or group YARN applications by a
configurable prefix.
h3. Proposed Implementation
* Add {{livy.server.app-name-prefix}} to {{LivyConf}}.
* In {{InteractiveSession.create()}}, use the configured prefix (if set)
instead of the hardcoded {{livy-session}} when building the default
{{spark.app.name}}.
* Preserve backward compatibility: when the property is not configured,
behavior is unchanged.
h3. Example
{code:bash}
# livy.conf
livy.server.app-name-prefix = my-cluster
# Create an interactive session
curl -X POST -H 'Content-Type: application/json' \
-d '{"kind":"spark"}' \
http://livy-host:8998/sessions
# YARN RM shows application name: my-cluster-3
yarn application -list | grep my-cluster
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
