[jira] [Commented] (LIVY-450) Override config options by environment variables.
[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475633#comment-16475633
]
Alexey Romanenko commented on LIVY-450:
---
PR: [https://github.com/apache/incubator-livy/pull/94]
> Override config options by environment variables.
> -
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
>Reporter: Alexey Romanenko
>Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (LIVY-450) Override config options by environment variables.
[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432095#comment-16432095
]
Alexey Romanenko commented on LIVY-450:
---
Since, there are no other thoughts on this, probably, it would make sense to go
forward and submit a PR. [~zjffdu] what do you think?
> Override config options by environment variables.
> -
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
>Reporter: Alexey Romanenko
>Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (LIVY-450) Override config options by environment variables.
[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421569#comment-16421569
]
Jeff Zhang commented on LIVY-450:
-
Sorry for late response, Thanks for the detail explanation [~ryanskraba]
[~aromanenko] The docker scenario make sense to me, thoughts ? \cc [~jerryshao]
> Override config options by environment variables.
> -
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
>Reporter: Alexey Romanenko
>Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (LIVY-450) Override config options by environment variables.
[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415723#comment-16415723
]
Ryan Skraba commented on LIVY-450:
--
Hello! Thanks for taking a look – I just wanted to add my voice why this is a
*good* feature for us.
In our system, we're launching the livy server in different ways, configured
for different types of user groups. We've had quite a bit of success deploying
these different scenarios with docker (and kubernetes), and it's much easier
for us to configure the *same* image *different* ways via environment
variables. To be clear – it's the admin, not users submitting jobs, that will
be setting these environment variables (via docker-compose or other) and
they're only read when the livy server starts.
If Spark is moving to deprecate environment variables, it's to simplify the
setup for the user submitting spark jobs – this is the right thing to do!
Usually, they'll be able to get all of the default spark configuration directly
from their cluster (or from the cluster admins) and it'll have reasonable
default values. Encapsulating the expertise from the admins in property files
makes this simpler for everyone.
A job server like Livy, serving many users, also shares this goal (simplifying
job submission for the user, encapsulating the cluster config from the
experts). Only the livy admin will be touching the livy.conf – or launching
the livy server process, with or without environment variables. (The blacklist
is kind of irrelevant here, since it applies to the user submitting jobs, not
the admin starting livy).
I hope this helps! There's a simple and clear workaround for us: to mount a
pre-configured livy.conf or modify the existing livy.conf during the image
entrypoint. Both of those feel relatively inelegant in comparison to
[~aromanenko]'s proposed solution, and a long-lived service like Livy, with a
single point of launch, can very likely justify the use of configuration via
environment.
> Override config options by environment variables.
> -
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
>Reporter: Alexey Romanenko
>Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (LIVY-450) Override config options by environment variables.
[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400514#comment-16400514
]
Alexey Romanenko commented on LIVY-450:
---
Our use case is the following - we have a docker image with Livy and other
projects, and this image can be used in different environment. So, depending on
where we run it, we have to set different config options. For now, it's mostly
_livy.server.recovery.mode_, _livy.spark.deploy-mode_ and _livy.spark.master._
So, it would be very useful to set these options by env variables without
changing the image or mounting different config.
> Override config options by environment variables.
> -
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
>Reporter: Alexey Romanenko
>Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (LIVY-450) Override config options by environment variables.
[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400417#comment-16400417
]
Jeff Zhang commented on LIVY-450:
-
Spark is trying to deprecate environment variable, I don't think it is
necessary for livy to support that. Could you describe in what kind of scenario
that you have to use environment variable instead of properties ?
> Override config options by environment variables.
> -
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
>Reporter: Alexey Romanenko
>Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (LIVY-450) Override config options by environment variables.
[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400409#comment-16400409
]
Alexey Romanenko commented on LIVY-450:
---
[~zjffdu] , thank you for your arguments, it makes sense for me but let me
argue a little bit with that.
1) If config option is blacklisted then we will ignore corresponding
environment variable as well and override only if this option is allowed.
2) Regarding that Spark prefers to use properties - yes, I totally agree with
that. On the other hand, why Livy can't support both ways of configuration? Is
there any principled objection against that which I probably missed?
> Override config options by environment variables.
> -
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
>Reporter: Alexey Romanenko
>Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (LIVY-450) Override config options by environment variables.
[
https://issues.apache.org/jira/browse/LIVY-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399683#comment-16399683
]
Jeff Zhang commented on LIVY-450:
-
I don't think it make sense to do that. Some properties are in blacklist which
don't allow user to change that. If you want to change that, you need to remove
it from blacklist instead of using env to override it. And IIUC, spark prefer
to use properties instead of env variable. Livy should also use this
convention.
> Override config options by environment variables.
> -
>
> Key: LIVY-450
> URL: https://issues.apache.org/jira/browse/LIVY-450
> Project: Livy
> Issue Type: Improvement
>Reporter: Alexey Romanenko
>Priority: Major
>
> Add a possibility to override all config options by environment variables.
> For example:
> {{LIVY_SPARK_MASTER}} automatically overrides {{livy.spark.master}} and
> {{LIVY_SPARK_DEPLOY_MODE}} overrides {{livy.spark.deploy-mode}}.
> It can be quite useful to use the same Livy image in different modes (local
> and cluster).
> Any pros/cons for or against that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
