[jira] [Commented] (AMBARI-22260) Update Spark2 log4j default settings to latest
[ https://issues.apache.org/jira/browse/AMBARI-22260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210628#comment-16210628 ] Saisai Shao commented on AMBARI-22260: -- Please help to review [~sumitmohanty] [~jluniya], thanks! > Update Spark2 log4j default settings to latest > -- > > Key: AMBARI-22260 > URL: https://issues.apache.org/jira/browse/AMBARI-22260 > Project: Ambari > Issue Type: Bug > Components: stacks >Affects Versions: 2.6.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > > Apache Spark log4j definition has changed in some places, we should update > this default setting in Ambari Spark2 stack definition to reflect such > changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AMBARI-22260) Update Spark2 log4j default settings to latest
Saisai Shao created AMBARI-22260: Summary: Update Spark2 log4j default settings to latest Key: AMBARI-22260 URL: https://issues.apache.org/jira/browse/AMBARI-22260 Project: Ambari Issue Type: Bug Components: stacks Affects Versions: 2.6.0 Reporter: Saisai Shao Assignee: Saisai Shao Apache Spark log4j definition has changed in some places, we should update this default setting in Ambari Spark2 stack definition to reflect such changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21226) Improve Spark thriftserver default configuration
[ https://issues.apache.org/jira/browse/AMBARI-21226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046344#comment-16046344 ] Saisai Shao commented on AMBARI-21226: -- CC [~sumitmohanty], please review, thanks! > Improve Spark thriftserver default configuration > > > Key: AMBARI-21226 > URL: https://issues.apache.org/jira/browse/AMBARI-21226 > Project: Ambari > Issue Type: Bug > Components: stacks >Reporter: Saisai Shao >Assignee: Saisai Shao > > Add two new default configurations to Ambari Spark stacks, this is mainly > used to improve the stability of long running of thriftserver. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AMBARI-21226) Improve Spark thriftserver default configuration
Saisai Shao created AMBARI-21226: Summary: Improve Spark thriftserver default configuration Key: AMBARI-21226 URL: https://issues.apache.org/jira/browse/AMBARI-21226 Project: Ambari Issue Type: Bug Components: stacks Reporter: Saisai Shao Assignee: Saisai Shao Add two new default configurations to Ambari Spark stacks, this is mainly used to improve the stability of long running of thriftserver. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AMBARI-21012) Livy service check fails with wire encryption setup
[
https://issues.apache.org/jira/browse/AMBARI-21012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saisai Shao reassigned AMBARI-21012:
Assignee: Saisai Shao
> Livy service check fails with wire encryption setup
> ---
>
> Key: AMBARI-21012
> URL: https://issues.apache.org/jira/browse/AMBARI-21012
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Yesha Vora
>Assignee: Saisai Shao
>
> Livy service check fails after enabling Wire encryption. This issue exist
> for Livy_server and Livy2_server both.
> STR:
> 1) Set up below properties in livy.conf to enable WE.
> {code}
> livy.ssl.trustStore
> livy.ssl.trustStorePassword
> livy.key-password
> livy.keystore
> livy.keystore.password {code}
> 2) Run Spark Service check.
> Spark service check with fail to validate Livy. It is using http port to
> connect to livy. When Wire encryption is enabled, it should use https
> protocol to connect to livy.
> {code:title=stderr}
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/service_check.py",
> line 62, in
> SparkServiceCheck().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 322, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/service_check.py",
> line 59, in service_check
> raise Fail(format("Connection to all Livy servers failed"))
> resource_management.core.exceptions.Fail: Connection to all Livy servers
> failed{code}
> {code:title=stdout}
> 2017-05-12 21:21:08,531 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2017-05-12 21:21:08,551 - Execute['kinit -kt
> /etc/security/keytabs/xxx.headless.keytab xxx@XXX; '] {'user': 'spark'}
> 2017-05-12 21:21:08,683 - Execute['kinit -kt
> /etc/security/keytabs/smokeuser.headless.keytab xxx@XXX; '] {'user': 'livy'}
> 2017-05-12 21:21:08,809 - Execute['curl -s -o /dev/null -w'%{http_code}'
> --negotiate -u: -k https://:18481 | grep 200'] {'logoutput': True,
> 'tries': 5, 'try_sleep': 3}
> 200
> 2017-05-12 21:21:09,010 - Execute['curl -s -o /dev/null -w'%{http_code}'
> --negotiate -u: -k http://:8999/sessions | grep 200'] {'logoutput': True,
> 'tries': 3, 'user': 'livy', 'try_sleep': 1}
> 2017-05-12 21:21:09,149 - Retrying after 1 seconds. Reason: Execution of
> 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k
> http://xxx:8999/sessions | grep 200' returned 1.
> 2017-05-12 21:21:10,286 - Retrying after 1 seconds. Reason: Execution of
> 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k
> http://:8999/sessions | grep 200' returned 1.
> Command failed after 1 tries{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AMBARI-21012) Livy service check fails with wire encryption setup
[
https://issues.apache.org/jira/browse/AMBARI-21012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011798#comment-16011798
]
Saisai Shao commented on AMBARI-21012:
--
[~sumitmohanty], please help to review, thanks!
> Livy service check fails with wire encryption setup
> ---
>
> Key: AMBARI-21012
> URL: https://issues.apache.org/jira/browse/AMBARI-21012
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Yesha Vora
>
> Livy service check fails after enabling Wire encryption. This issue exist
> for Livy_server and Livy2_server both.
> STR:
> 1) Set up below properties in livy.conf to enable WE.
> {code}
> livy.ssl.trustStore
> livy.ssl.trustStorePassword
> livy.key-password
> livy.keystore
> livy.keystore.password {code}
> 2) Run Spark Service check.
> Spark service check with fail to validate Livy. It is using http port to
> connect to livy. When Wire encryption is enabled, it should use https
> protocol to connect to livy.
> {code:title=stderr}
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/service_check.py",
> line 62, in
> SparkServiceCheck().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 322, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/service_check.py",
> line 59, in service_check
> raise Fail(format("Connection to all Livy servers failed"))
> resource_management.core.exceptions.Fail: Connection to all Livy servers
> failed{code}
> {code:title=stdout}
> 2017-05-12 21:21:08,531 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2017-05-12 21:21:08,551 - Execute['kinit -kt
> /etc/security/keytabs/xxx.headless.keytab xxx@XXX; '] {'user': 'spark'}
> 2017-05-12 21:21:08,683 - Execute['kinit -kt
> /etc/security/keytabs/smokeuser.headless.keytab xxx@XXX; '] {'user': 'livy'}
> 2017-05-12 21:21:08,809 - Execute['curl -s -o /dev/null -w'%{http_code}'
> --negotiate -u: -k https://:18481 | grep 200'] {'logoutput': True,
> 'tries': 5, 'try_sleep': 3}
> 200
> 2017-05-12 21:21:09,010 - Execute['curl -s -o /dev/null -w'%{http_code}'
> --negotiate -u: -k http://:8999/sessions | grep 200'] {'logoutput': True,
> 'tries': 3, 'user': 'livy', 'try_sleep': 1}
> 2017-05-12 21:21:09,149 - Retrying after 1 seconds. Reason: Execution of
> 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k
> http://xxx:8999/sessions | grep 200' returned 1.
> 2017-05-12 21:21:10,286 - Retrying after 1 seconds. Reason: Execution of
> 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k
> http://:8999/sessions | grep 200' returned 1.
> Command failed after 1 tries{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (AMBARI-19997) Fix file permission issue of Livy recovery folder in Ambari
Saisai Shao created AMBARI-19997: Summary: Fix file permission issue of Livy recovery folder in Ambari Key: AMBARI-19997 URL: https://issues.apache.org/jira/browse/AMBARI-19997 Project: Ambari Issue Type: Bug Affects Versions: 2.5.0 Reporter: Saisai Shao Assignee: Saisai Shao -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMBARI-19577) Add Livy session recovery configurations in Ambari
[ https://issues.apache.org/jira/browse/AMBARI-19577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825733#comment-15825733 ] Saisai Shao commented on AMBARI-19577: -- CC [~sumitmohanty], please help to review, thanks a lot. > Add Livy session recovery configurations in Ambari > -- > > Key: AMBARI-19577 > URL: https://issues.apache.org/jira/browse/AMBARI-19577 > Project: Ambari > Issue Type: Bug > Components: stacks >Affects Versions: 2.5.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > > Update the Ambari Livy configurations to support session recovery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-19577) Add Livy session recovery configurations in Ambari
Saisai Shao created AMBARI-19577: Summary: Add Livy session recovery configurations in Ambari Key: AMBARI-19577 URL: https://issues.apache.org/jira/browse/AMBARI-19577 Project: Ambari Issue Type: Bug Components: stacks Affects Versions: 2.5.0 Reporter: Saisai Shao Assignee: Saisai Shao Update the Ambari Livy configurations to support session recovery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-17489) Remove spark.yarn.max.executor.failures configuration in Spark Ambari definition
[ https://issues.apache.org/jira/browse/AMBARI-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590615#comment-15590615 ] Saisai Shao commented on AMBARI-17489: -- [~sumitmohanty], I've rebased this patch to branch-2.4 and 2.5 (https://reviews.apache.org/r/49484/), looks like branch 2.5 can be auto merged, so one patch is enough. Please review. Thanks a lot. > Remove spark.yarn.max.executor.failures configuration in Spark Ambari > definition > > > Key: AMBARI-17489 > URL: https://issues.apache.org/jira/browse/AMBARI-17489 > Project: Ambari > Issue Type: Bug > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.4.2 > > > Currently spark.yarn.max.executor.failures should be configured in Ambari web > UI, which makes it inflexible to change, actually the default default in > Spark is more reasonable than here as "3", so propose to remove this > configuration starting from Spark 1.6.0. > CC [~zjffdu], please suggest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-18091) Use https url for Spark2 Service check when WireEncryption is enabled
[ https://issues.apache.org/jira/browse/AMBARI-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414751#comment-15414751 ] Saisai Shao commented on AMBARI-18091: -- Please help to review, [~sumitmohanty] [~jluniya], thanks a lot. > Use https url for Spark2 Service check when WireEncryption is enabled > - > > Key: AMBARI-18091 > URL: https://issues.apache.org/jira/browse/AMBARI-18091 > Project: Ambari > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Yesha Vora >Assignee: Saisai Shao >Priority: Critical > > When Wire encryption is enabled, https://:/ should be used > as spark2 history server url. > Port 18481 can be used as default port for spark2 History. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AMBARI-18091) Use https url for Spark2 Service check when WireEncryption is enabled
[ https://issues.apache.org/jira/browse/AMBARI-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned AMBARI-18091: Assignee: Saisai Shao > Use https url for Spark2 Service check when WireEncryption is enabled > - > > Key: AMBARI-18091 > URL: https://issues.apache.org/jira/browse/AMBARI-18091 > Project: Ambari > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Yesha Vora >Assignee: Saisai Shao >Priority: Critical > > When Wire encryption is enabled, https://:/ should be used > as spark2 history server url. > Port 18481 can be used as default port for spark2 History. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-17954) Fix Spark hdp.version issues in upgrading and fresh install
[ https://issues.apache.org/jira/browse/AMBARI-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398740#comment-15398740 ] Saisai Shao commented on AMBARI-17954: -- CC [~sumitmohanty] [~jluniya], please help to review, thanks a lot. > Fix Spark hdp.version issues in upgrading and fresh install > --- > > Key: AMBARI-17954 > URL: https://issues.apache.org/jira/browse/AMBARI-17954 > Project: Ambari > Issue Type: Bug > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > > Ambari Spark definitions have several issues related to hdp.version, so here > fix it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-17954) Fix Spark hdp.version issues in upgrading and fresh install
Saisai Shao created AMBARI-17954: Summary: Fix Spark hdp.version issues in upgrading and fresh install Key: AMBARI-17954 URL: https://issues.apache.org/jira/browse/AMBARI-17954 Project: Ambari Issue Type: Bug Components: stacks Affects Versions: 2.4.0 Reporter: Saisai Shao Assignee: Saisai Shao Ambari Spark definitions have several issues related to hdp.version, so here fix it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-16864) Add unit tests for Spark2 service definition
[ https://issues.apache.org/jira/browse/AMBARI-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383435#comment-15383435 ] Saisai Shao commented on AMBARI-16864: -- Done, patch updated. > Add unit tests for Spark2 service definition > > > Key: AMBARI-16864 > URL: https://issues.apache.org/jira/browse/AMBARI-16864 > Project: Ambari > Issue Type: New Feature > Components: stacks >Affects Versions: 2.4.0 >Reporter: Jayush Luniya >Assignee: Saisai Shao > Fix For: 2.5.0 > > > Propose to add Spark2 in Ambari as a standalone service in parallel with > Spark1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-17545) Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading
[ https://issues.apache.org/jira/browse/AMBARI-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360862#comment-15360862 ] Saisai Shao commented on AMBARI-17545: -- [~sumitmohanty] [~afernandez] please help to review, not sure is it enough to address the upgrading issue. > Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions > property in upgrading > - > > Key: AMBARI-17545 > URL: https://issues.apache.org/jira/browse/AMBARI-17545 > Project: Ambari > Issue Type: Bug > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > > These two properties are not used for Spark, set to empty is only for > backward compatibility. So removing these two configurations starting from > HDP 2.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-17545) Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading
Saisai Shao created AMBARI-17545: Summary: Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading Key: AMBARI-17545 URL: https://issues.apache.org/jira/browse/AMBARI-17545 Project: Ambari Issue Type: Bug Components: stacks Affects Versions: 2.4.0 Reporter: Saisai Shao Assignee: Saisai Shao These two properties are not used for Spark, set to empty is only for backward compatibility. So removing these two configurations starting from HDP 2.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-17489) Remove spark.yarn.max.executor.failures configuration in Spark Ambari definition
Saisai Shao created AMBARI-17489: Summary: Remove spark.yarn.max.executor.failures configuration in Spark Ambari definition Key: AMBARI-17489 URL: https://issues.apache.org/jira/browse/AMBARI-17489 Project: Ambari Issue Type: Bug Components: stacks Affects Versions: 2.4.0 Reporter: Saisai Shao Assignee: Saisai Shao Currently spark.yarn.max.executor.failures should be configured in Ambari web UI, which makes it inflexible to change, actually the default default in Spark is more reasonable than here as "3", so propose to remove this configuration starting from Spark 1.6.0. CC [~zjffdu], please suggest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-17459) remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in HDP2.5 definition
Saisai Shao created AMBARI-17459: Summary: remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in HDP2.5 definition Key: AMBARI-17459 URL: https://issues.apache.org/jira/browse/AMBARI-17459 Project: Ambari Issue Type: Bug Components: stacks Affects Versions: 2.4.0 Reporter: Saisai Shao Assignee: Saisai Shao These two properties are not used for Spark, set to empty is only for backward compatibility. So removing these two configurations starting from HDP 2.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AMBARI-17328) Spark2 history server quick link still point to spark1 url
[ https://issues.apache.org/jira/browse/AMBARI-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-17328: - Issue Type: Sub-task (was: Technical task) > Spark2 history server quick link still point to spark1 url > -- > > Key: AMBARI-17328 > URL: https://issues.apache.org/jira/browse/AMBARI-17328 > Project: Ambari > Issue Type: Sub-task > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-17391) Spark thriftserver fails to start when umask = 027 due to permission issues on java-opts
[ https://issues.apache.org/jira/browse/AMBARI-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345864#comment-15345864 ] Saisai Shao commented on AMBARI-17391: -- Hi [~jluniya], I think we should also has such change in the SPARK2 definition. > Spark thriftserver fails to start when umask = 027 due to permission issues > on java-opts > - > > Key: AMBARI-17391 > URL: https://issues.apache.org/jira/browse/AMBARI-17391 > Project: Ambari > Issue Type: Bug > Components: stacks >Affects Versions: 2.4.0 >Reporter: Jayush Luniya >Assignee: Jayush Luniya >Priority: Critical > Fix For: 2.4.0 > > Attachments: AMBARI-17391.patch > > > AMBARI-16651 added java-opts config file. SparkThriftServer is started as > hive/hadoop user whereas java-opts file is created as spark/spark user. When > umask is set to 0027 STS fails to start due to permission issues on > java-opts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-17328) Spark2 history server quick link still point to spark1 url
[ https://issues.apache.org/jira/browse/AMBARI-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340981#comment-15340981 ] Saisai Shao commented on AMBARI-17328: -- CC [~jluniya] [~zjffdu] please help to review. > Spark2 history server quick link still point to spark1 url > -- > > Key: AMBARI-17328 > URL: https://issues.apache.org/jira/browse/AMBARI-17328 > Project: Ambari > Issue Type: Technical task > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-17328) Spark2 history server quick link still point to spark1 url
Saisai Shao created AMBARI-17328: Summary: Spark2 history server quick link still point to spark1 url Key: AMBARI-17328 URL: https://issues.apache.org/jira/browse/AMBARI-17328 Project: Ambari Issue Type: Technical task Components: stacks Affects Versions: 2.4.0 Reporter: Saisai Shao Assignee: Saisai Shao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-17204) Fix bug in for Spark2 thrift server definition
[ https://issues.apache.org/jira/browse/AMBARI-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327859#comment-15327859 ] Saisai Shao commented on AMBARI-17204: -- CC [~zjffdu], please help to review. > Fix bug in for Spark2 thrift server definition > -- > > Key: AMBARI-17204 > URL: https://issues.apache.org/jira/browse/AMBARI-17204 > Project: Ambari > Issue Type: Sub-task > Components: stacks >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.4.0 > > > Currently because of some issues, fairscheduler xml file cannot be generated > by Ambari, here fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AMBARI-17204) Fix bug in for Spark2 thrift server definition
[ https://issues.apache.org/jira/browse/AMBARI-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-17204: - Attachment: (was: AMBARI-17204.patch) > Fix bug in for Spark2 thrift server definition > -- > > Key: AMBARI-17204 > URL: https://issues.apache.org/jira/browse/AMBARI-17204 > Project: Ambari > Issue Type: Sub-task > Components: stacks >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.4.0 > > > Currently because of some issues, fairscheduler xml file cannot be generated > by Ambari, here fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AMBARI-17204) Fix bug in for Spark2 thrift server definition
[ https://issues.apache.org/jira/browse/AMBARI-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-17204: - Attachment: AMBARI-17204.patch > Fix bug in for Spark2 thrift server definition > -- > > Key: AMBARI-17204 > URL: https://issues.apache.org/jira/browse/AMBARI-17204 > Project: Ambari > Issue Type: Sub-task > Components: stacks >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.4.0 > > Attachments: AMBARI-17204.patch > > > Currently because of some issues, fairscheduler xml file cannot be generated > by Ambari, here fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AMBARI-17204) Fix bug in for Spark2 thrift server definition
[ https://issues.apache.org/jira/browse/AMBARI-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned AMBARI-17204: Assignee: Saisai Shao > Fix bug in for Spark2 thrift server definition > -- > > Key: AMBARI-17204 > URL: https://issues.apache.org/jira/browse/AMBARI-17204 > Project: Ambari > Issue Type: Sub-task > Components: stacks >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.4.0 > > > Currently because of some issues, fairscheduler xml file cannot be generated > by Ambari, here fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-17204) Fix bug in for Spark2 thrift server definition
Saisai Shao created AMBARI-17204: Summary: Fix bug in for Spark2 thrift server definition Key: AMBARI-17204 URL: https://issues.apache.org/jira/browse/AMBARI-17204 Project: Ambari Issue Type: Sub-task Reporter: Saisai Shao Currently because of some issues, fairscheduler xml file cannot be generated by Ambari, here fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-16864) Add unit tests for Spark2 service definition
[ https://issues.apache.org/jira/browse/AMBARI-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327702#comment-15327702 ] Saisai Shao commented on AMBARI-16864: -- Patch is submitted, please review [~jluniya], thanks a lot. > Add unit tests for Spark2 service definition > > > Key: AMBARI-16864 > URL: https://issues.apache.org/jira/browse/AMBARI-16864 > Project: Ambari > Issue Type: New Feature > Components: stacks >Affects Versions: 2.4.0 >Reporter: Jayush Luniya >Assignee: Saisai Shao > Fix For: 2.4.0 > > > Propose to add Spark2 in Ambari as a standalone service in parallel with > Spark1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-16864) Add unit tests for Spark2 service definition
[ https://issues.apache.org/jira/browse/AMBARI-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306170#comment-15306170 ] Saisai Shao commented on AMBARI-16864: -- Hi [~jluniya], two questions about unit test: 1. How do I verify the unit test locally? 2. How to get the config file like "configs/default.json" which has Spark2 definitions? Thanks a lot. > Add unit tests for Spark2 service definition > > > Key: AMBARI-16864 > URL: https://issues.apache.org/jira/browse/AMBARI-16864 > Project: Ambari > Issue Type: New Feature > Components: stacks >Affects Versions: 2.4.0 >Reporter: Jayush Luniya >Assignee: Saisai Shao > Fix For: 2.4.0 > > > Propose to add Spark2 in Ambari as a standalone service in parallel with > Spark1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-16864) Add unit tests for Spark2 service definition
[ https://issues.apache.org/jira/browse/AMBARI-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299704#comment-15299704 ] Saisai Shao commented on AMBARI-16864: -- Thanks [~jluniya], I will add some unit tests about it. > Add unit tests for Spark2 service definition > > > Key: AMBARI-16864 > URL: https://issues.apache.org/jira/browse/AMBARI-16864 > Project: Ambari > Issue Type: New Feature > Components: stacks >Affects Versions: 2.4.0 >Reporter: Jayush Luniya >Assignee: Saisai Shao > Fix For: 2.4.0 > > > Propose to add Spark2 in Ambari as a standalone service in parallel with > Spark1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AMBARI-16759) Remove useless Spark definitions
[ https://issues.apache.org/jira/browse/AMBARI-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-16759: - Description: Remove useless log dir definition in Spark service definition. In Spark service definition, there're two log related definitions: one is spark_log_dir which is pointed to /var/log/spark/log another is spark_logs_dir which is pointed to /log The latter is never used, so here propose to remove it. Also remove useless tez conf in Spark service definition was: Remove useless log dir definition in Spark service definition. In Spark service definition, there're two log related definitions: one is spark_log_dir which is pointed to /var/log/spark/log another is spark_logs_dir which is pointed to /log The latter is never used, so here propose to remove it. > Remove useless Spark definitions > > > Key: AMBARI-16759 > URL: https://issues.apache.org/jira/browse/AMBARI-16759 > Project: Ambari > Issue Type: Improvement > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao >Priority: Minor > > Remove useless log dir definition in Spark service definition. > In Spark service definition, there're two log related definitions: > one is spark_log_dir which is pointed to /var/log/spark/log > another is spark_logs_dir which is pointed to /log > The latter is never used, so here propose to remove it. > Also remove useless tez conf in Spark service definition -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AMBARI-16759) Remove useless Spark definitions
[ https://issues.apache.org/jira/browse/AMBARI-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-16759: - Summary: Remove useless Spark definitions (was: Remove useless log dir definition in Spark Ambari definition) > Remove useless Spark definitions > > > Key: AMBARI-16759 > URL: https://issues.apache.org/jira/browse/AMBARI-16759 > Project: Ambari > Issue Type: Improvement > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao >Priority: Minor > > Remove useless log dir definition in Spark service definition. > In Spark service definition, there're two log related definitions: > one is spark_log_dir which is pointed to /var/log/spark/log > another is spark_logs_dir which is pointed to /log > The latter is never used, so here propose to remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-16759) Remove useless log dir definition in Spark Ambari definition
Saisai Shao created AMBARI-16759: Summary: Remove useless log dir definition in Spark Ambari definition Key: AMBARI-16759 URL: https://issues.apache.org/jira/browse/AMBARI-16759 Project: Ambari Issue Type: Improvement Components: stacks Affects Versions: 2.4.0 Reporter: Saisai Shao Assignee: Saisai Shao Priority: Minor Remove useless log dir definition in Spark service definition. In Spark service definition, there're two log related definitions: one is spark_log_dir which is pointed to /var/log/spark/log another is spark_logs_dir which is pointed to /log The latter is never used, so here propose to remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMBARI-16753) Add new Spark2 service definition to Ambari
[ https://issues.apache.org/jira/browse/AMBARI-16753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290510#comment-15290510 ] Saisai Shao commented on AMBARI-16753: -- CC [~sumitmohanty] [~jluniya], please help to review, thanks a lot. > Add new Spark2 service definition to Ambari > --- > > Key: AMBARI-16753 > URL: https://issues.apache.org/jira/browse/AMBARI-16753 > Project: Ambari > Issue Type: New Feature > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > > Propose to add Spark2 in Ambari as a standalone service in parallel with > Spark1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AMBARI-16753) Add new Spark2 service definition to Ambari
[ https://issues.apache.org/jira/browse/AMBARI-16753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned AMBARI-16753: Assignee: Saisai Shao > Add new Spark2 service definition to Ambari > --- > > Key: AMBARI-16753 > URL: https://issues.apache.org/jira/browse/AMBARI-16753 > Project: Ambari > Issue Type: New Feature > Components: stacks >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao > > Propose to add Spark2 in Ambari as a standalone service in parallel with > Spark1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AMBARI-16753) Add new Spark2 service definition to Ambari
Saisai Shao created AMBARI-16753: Summary: Add new Spark2 service definition to Ambari Key: AMBARI-16753 URL: https://issues.apache.org/jira/browse/AMBARI-16753 Project: Ambari Issue Type: New Feature Components: stacks Affects Versions: 2.4.0 Reporter: Saisai Shao Propose to add Spark2 in Ambari as a standalone service in parallel with Spark1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
