Re: Review Request 62925: Livy protocol to be set to https in Zeppelin's interpreter setting
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62925/#review187756 --- ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/params.py Lines 230 (patched) <https://reviews.apache.org/r/62925/#comment264805> livy.key-password is optional, it is better to check livy.keystore - Jeff Zhang On Oct. 12, 2017, 7:06 a.m., Prabhjyot Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62925/ > --- > > (Updated Oct. 12, 2017, 7:06 a.m.) > > > Review request for Ambari, DIPAYAN BHOWMICK, Jaimin Jetly, Jayush Luniya, > Rohit Choudhary, Sumit Mohanty, and venkat sairam. > > > Bugs: AMBARI-22214 > https://issues.apache.org/jira/browse/AMBARI-22214 > > > Repository: ambari > > > Description > --- > > If Livy is running on https mode then Livy protocol to be set to https in > Zeppelin's interpreter setting. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/master.py > 09944bd277 > > ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/params.py > 3242f2621a > > > Diff: https://reviews.apache.org/r/62925/diff/1/ > > > Testing > --- > > Manually > > > Thanks, > > Prabhjyot Singh > >
Re: Review Request 62461: Livy not working with the default configuration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62461/#review185949 --- Ship it! Ship It! - Jeff Zhang On Sept. 21, 2017, 1 p.m., Prabhjyot Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62461/ > --- > > (Updated Sept. 21, 2017, 1 p.m.) > > > Review request for Ambari, DIPAYAN BHOWMICK, Jaimin Jetly, Jayush Luniya, > Rohit Choudhary, Sumit Mohanty, and venkat sairam. > > > Bugs: AMBARI-22021 > https://issues.apache.org/jira/browse/AMBARI-22021 > > > Repository: ambari > > > Description > --- > > This started failing with ZEPPELIN-2909, which introduces `"className": > "org.apache.zeppelin.livy.LivySharedInterpreter"` and now Livy fails with > following exception; > > > java.lang.NullPointerException > at > org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:109) > at > org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) > at > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493) > at org.apache.zeppelin.scheduler.Job.run(Job.java:175) > at > org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/interpreter_json_template.py > 6a9891972b > > ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/livy2_config_template.py > 71d381705a > > > Diff: https://reviews.apache.org/r/62461/diff/3/ > > > Testing > --- > > Manually > > > Thanks, > > Prabhjyot Singh > >
Re: Review Request 62461: Livy not working with the default configuration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62461/#review185886 --- ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/interpreter_json_template.py Line 250 (original), 255 (patched) <https://reviews.apache.org/r/62461/#comment262224> It should be still `scoped` ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/livy2_config_template.py Line 101 (original), 106 (patched) <https://reviews.apache.org/r/62461/#comment262225> still scoped - Jeff Zhang On Sept. 21, 2017, 12:54 p.m., Prabhjyot Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62461/ > --- > > (Updated Sept. 21, 2017, 12:54 p.m.) > > > Review request for Ambari, DIPAYAN BHOWMICK, Jaimin Jetly, Jayush Luniya, > Rohit Choudhary, Sumit Mohanty, and venkat sairam. > > > Bugs: AMBARI-22021 > https://issues.apache.org/jira/browse/AMBARI-22021 > > > Repository: ambari > > > Description > --- > > This started failing with ZEPPELIN-2909, which introduces `"className": > "org.apache.zeppelin.livy.LivySharedInterpreter"` and now Livy fails with > following exception; > > > java.lang.NullPointerException > at > org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:109) > at > org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) > at > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493) > at org.apache.zeppelin.scheduler.Job.run(Job.java:175) > at > org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/interpreter_json_template.py > 6a9891972b > > ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/livy2_config_template.py > 71d381705a > > > Diff: https://reviews.apache.org/r/62461/diff/2/ > > > Testing > --- > > Manually > > > Thanks, > > Prabhjyot Singh > >
Review Request 57906: AMBARI-20556. Add SPARK_CONF_DIR in livy-env when upgrading from HDP 2.5 to 2.6
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57906/ --- Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. Bugs: AMBARI-20556 https://issues.apache.org/jira/browse/AMBARI-20556 Repository: ambari Description --- Should add SPARK_CONF_DIR in livy-env when upgrading from HDP 2.5 to 2.6 Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 19b55ec ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 063b362 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml e637c4c Diff: https://reviews.apache.org/r/57906/diff/1/ Testing --- Test upgrading from HDP 2.5 to 2.6 manually Thanks, Jeff Zhang
Re: Review Request 55486: AMBARI-19417. Enable hiveContext in Livy
> On Jan. 13, 2017, 6:51 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-conf.xml, > > line 79 > > <https://reviews.apache.org/r/55486/diff/1/?file=1604394#file1604394line79> > > > > This a new stack, please everywhere @afernandez, when will you commi it ? Thanks - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55486/#review161561 --- On Jan. 14, 2017, 12:51 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55486/ > --- > > (Updated Jan. 14, 2017, 12:51 a.m.) > > > Review request for Ambari and Alejandro Fernandez. > > > Bugs: AMBARI-19417 > https://issues.apache.org/jira/browse/AMBARI-19417 > > > Repository: ambari > > > Description > --- > > Update the livy configuration to enable hiveContext > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-conf.xml > b7bfa73 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-env.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/livy2-conf.xml > 457869d > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/livy2-env.xml > afb1d17 > > Diff: https://reviews.apache.org/r/55486/diff/ > > > Testing > --- > > Tested maunally. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 55486: AMBARI-19417. Enable hiveContext in Livy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55486/ --- (Updated Jan. 14, 2017, 12:51 a.m.) Review request for Ambari and Alejandro Fernandez. Changes --- Address comment Bugs: AMBARI-19417 https://issues.apache.org/jira/browse/AMBARI-19417 Repository: ambari Description --- Update the livy configuration to enable hiveContext Diffs (updated) - ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-conf.xml b7bfa73 ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/livy2-conf.xml 457869d ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/livy2-env.xml afb1d17 Diff: https://reviews.apache.org/r/55486/diff/ Testing --- Tested maunally. Thanks, Jeff Zhang
Review Request 55486: AMBARI-19417. Enable hiveContext in Livy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55486/ --- Review request for Ambari and Alejandro Fernandez. Bugs: AMBARI-19417 https://issues.apache.org/jira/browse/AMBARI-19417 Repository: ambari Description --- Update the livy configuration to enable hiveContext Diffs - ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-conf.xml b7bfa73 ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/livy2-conf.xml 457869d ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/livy2-env.xml afb1d17 Diff: https://reviews.apache.org/r/55486/diff/ Testing --- Tested maunally. Thanks, Jeff Zhang
Re: Review Request 54471: AMBARI-19025. Add livy.spark.master to livy.conf and update spark-blacklist.conf
> On Dec. 7, 2016, 5:43 a.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-conf.xml, > > line 21 > > <https://reviews.apache.org/r/54471/diff/1/?file=1578511#file1578511line21> > > > > If these are new config types - should they be in the metainfo.xml too? > > In the section? They are not new configuratin files. Just add livy.spark.master to livy.conf and also update content of spark-blacklist.conf - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54471/#review158307 --- On Dec. 7, 2016, 5:04 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54471/ > --- > > (Updated Dec. 7, 2016, 5:04 a.m.) > > > Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. > > > Bugs: AMBARI-19025 > https://issues.apache.org/jira/browse/AMBARI-19025 > > > Repository: ambari > > > Description > --- > > We would use livy 0.3 in HDP 2.6. and need to add livy.spark.master to > livy.conf and also update spark-blacklist.con > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-conf.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/livy-spark-blacklist.xml > PRE-CREATION > > Diff: https://reviews.apache.org/r/54471/diff/ > > > Testing > --- > > Manully tested in HDP 2.6 > > > Thanks, > > Jeff Zhang > >
Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53334/ --- (Updated Nov. 8, 2016, 12:05 a.m.) Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sumit Mohanty. Bugs: AMBARI-18758 https://issues.apache.org/jira/browse/AMBARI-18758 Repository: ambari Description --- 2 configuration key are changed. livy.server.kerberos.keytab --> livy.server.launch.kerberos.keytab livy.server.kerberos.principal --> livy.server.launch.kerberos.principal Diffs (updated) - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 3a1c482 ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json PRE-CREATION Diff: https://reviews.apache.org/r/53334/diff/ Testing --- I have verified it in HDP 2.6. But havn't done for the upgrade. @afernandez, do you know how to change the configuration key for upgrade ? I didn't find samples in config-upgrade.xml Thanks, Jeff Zhang
Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53334/ --- (Updated Nov. 4, 2016, 6:04 a.m.) Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sumit Mohanty. Bugs: AMBARI-18758 https://issues.apache.org/jira/browse/AMBARI-18758 Repository: ambari Description --- 2 configuration key are changed. livy.server.kerberos.keytab --> livy.server.launch.kerberos.keytab livy.server.kerberos.principal --> livy.server.launch.kerberos.principal Diffs (updated) - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 3a1c482 ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json PRE-CREATION Diff: https://reviews.apache.org/r/53334/diff/ Testing --- I have verified it in HDP 2.6. But havn't done for the upgrade. @afernandez, do you know how to change the configuration key for upgrade ? I didn't find samples in config-upgrade.xml Thanks, Jeff Zhang
Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade
> On Nov. 3, 2016, 1:21 a.m., Sumit Mohanty wrote: > > Lets have a bug open to fix it in the upgrade pack. > > > > Pls talk to Robert Levas and see if the persisted kerberos.json need to be > > modified or it will get auto modified when stack is upgraded. Thanks Sumit, create https://issues.apache.org/jira/browse/AMBARI-18780 for upgrade fix. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53334/#review154666 --- On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53334/ > --- > > (Updated Nov. 1, 2016, 1:52 a.m.) > > > Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. > > > Bugs: AMBARI-18758 > https://issues.apache.org/jira/browse/AMBARI-18758 > > > Repository: ambari > > > Description > --- > > 2 configuration key are changed. > > livy.server.kerberos.keytab --> livy.server.launch.kerberos.keytab > livy.server.kerberos.principal --> livy.server.launch.kerberos.principal > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > 3a1c482 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/53334/diff/ > > > Testing > --- > > I have verified it in HDP 2.6. But havn't done for the upgrade. > > @afernandez, do you know how to change the configuration key for upgrade ? I > didn't find samples in config-upgrade.xml > > > Thanks, > > Jeff Zhang > >
Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade
> On Nov. 1, 2016, 5:36 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py, > > line 234 > > <https://reviews.apache.org/r/53334/diff/1/?file=1550503#file1550503line234> > > > > Is this meant to account for a different Spark version in HDP stack? > > If so, we can use stack_features.json and the check_stack_feature > > function to make it more clear why this is the case. > > Jeff Zhang wrote: > yes, this is only for HDP 2.6 or later version. This is not feature, > just a compablity break, so I think it is not necessary to put it in > stack_features.json > > Jeff Zhang wrote: > BTW, do you know how to change the configuration key for upgrade ? I > didn't find samples in config-upgrade.xml > > Alejandro Fernandez wrote: > We still don't have upgrade packs and config packs for HDP 2.5 to 2.6 Do you mean currently upgrade from 2.5 to 2.6 is not supported, and I can add that to config-upgrade.xml when it is supported ? - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53334/#review154422 ------- On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53334/ > --- > > (Updated Nov. 1, 2016, 1:52 a.m.) > > > Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. > > > Bugs: AMBARI-18758 > https://issues.apache.org/jira/browse/AMBARI-18758 > > > Repository: ambari > > > Description > --- > > 2 configuration key are changed. > > livy.server.kerberos.keytab --> livy.server.launch.kerberos.keytab > livy.server.kerberos.principal --> livy.server.launch.kerberos.principal > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > 3a1c482 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/53334/diff/ > > > Testing > --- > > I have verified it in HDP 2.6. But havn't done for the upgrade. > > @afernandez, do you know how to change the configuration key for upgrade ? I > didn't find samples in config-upgrade.xml > > > Thanks, > > Jeff Zhang > >
Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade
> On Nov. 1, 2016, 5:36 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py, > > line 234 > > <https://reviews.apache.org/r/53334/diff/1/?file=1550503#file1550503line234> > > > > Is this meant to account for a different Spark version in HDP stack? > > If so, we can use stack_features.json and the check_stack_feature > > function to make it more clear why this is the case. > > Jeff Zhang wrote: > yes, this is only for HDP 2.6 or later version. This is not feature, > just a compablity break, so I think it is not necessary to put it in > stack_features.json BTW, do you know how to change the configuration key for upgrade ? I didn't find samples in config-upgrade.xml - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53334/#review154422 --- On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53334/ > --- > > (Updated Nov. 1, 2016, 1:52 a.m.) > > > Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. > > > Bugs: AMBARI-18758 > https://issues.apache.org/jira/browse/AMBARI-18758 > > > Repository: ambari > > > Description > --- > > 2 configuration key are changed. > > livy.server.kerberos.keytab --> livy.server.launch.kerberos.keytab > livy.server.kerberos.principal --> livy.server.launch.kerberos.principal > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > 3a1c482 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/53334/diff/ > > > Testing > --- > > I have verified it in HDP 2.6. But havn't done for the upgrade. > > @afernandez, do you know how to change the configuration key for upgrade ? I > didn't find samples in config-upgrade.xml > > > Thanks, > > Jeff Zhang > >
Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade
> On Nov. 1, 2016, 5:36 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py, > > line 234 > > <https://reviews.apache.org/r/53334/diff/1/?file=1550503#file1550503line234> > > > > Is this meant to account for a different Spark version in HDP stack? > > If so, we can use stack_features.json and the check_stack_feature > > function to make it more clear why this is the case. yes, this is only for HDP 2.6 or later version. This is not feature, just a compablity break, so I think it is not necessary to put it in stack_features.json - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53334/#review154422 ------- On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53334/ > --- > > (Updated Nov. 1, 2016, 1:52 a.m.) > > > Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. > > > Bugs: AMBARI-18758 > https://issues.apache.org/jira/browse/AMBARI-18758 > > > Repository: ambari > > > Description > --- > > 2 configuration key are changed. > > livy.server.kerberos.keytab --> livy.server.launch.kerberos.keytab > livy.server.kerberos.principal --> livy.server.launch.kerberos.principal > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > 3a1c482 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/53334/diff/ > > > Testing > --- > > I have verified it in HDP 2.6. But havn't done for the upgrade. > > @afernandez, do you know how to change the configuration key for upgrade ? I > didn't find samples in config-upgrade.xml > > > Thanks, > > Jeff Zhang > >
Review Request 53334: AMBARI-18758. Livy configuration change for upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53334/ --- Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. Bugs: AMBARI-18758 https://issues.apache.org/jira/browse/AMBARI-18758 Repository: ambari Description --- 2 configuration key are changed. livy.server.kerberos.keytab --> livy.server.launch.kerberos.keytab livy.server.kerberos.principal --> livy.server.launch.kerberos.principal Diffs - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 3a1c482 ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json PRE-CREATION Diff: https://reviews.apache.org/r/53334/diff/ Testing --- I have verified it in HDP 2.6. But havn't done for the upgrade. Thanks, Jeff Zhang
Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53334/ --- (Updated Nov. 1, 2016, 1:52 a.m.) Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. Bugs: AMBARI-18758 https://issues.apache.org/jira/browse/AMBARI-18758 Repository: ambari Description --- 2 configuration key are changed. livy.server.kerberos.keytab --> livy.server.launch.kerberos.keytab livy.server.kerberos.principal --> livy.server.launch.kerberos.principal Diffs - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 3a1c482 ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json PRE-CREATION Diff: https://reviews.apache.org/r/53334/diff/ Testing (updated) --- I have verified it in HDP 2.6. But havn't done for the upgrade. @afernandez, do you know how to change the configuration key for upgrade ? I didn't find samples in config-upgrade.xml Thanks, Jeff Zhang
Re: Review Request 51078: AMBARI-18142. Define keytab/principal for Spark Thrift Server
> On Aug. 15, 2016, 4:15 a.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-env.xml, > > line 76 > > <https://reviews.apache.org/r/51078/diff/1/?file=1472662#file1472662line76> > > > > OK. Can you verify with an upgrade test on a secured cluster? If you > > have already done so then great. Do you mean ambari upgrade ? But how can I do ambari grade test without committing this ? - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51078/#review145737 ------- On Aug. 14, 2016, 3:45 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51078/ > --- > > (Updated Aug. 14, 2016, 3:45 a.m.) > > > Review request for Ambari and Sumit Mohanty. > > > Bugs: AMBARI-18142 > https://issues.apache.org/jira/browse/AMBARI-18142 > > > Repository: ambari > > > Description > --- > > Should define keytab/principle for spark thrift server in its configuration > file 'spark-thrift-sparkconf' > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-env.xml > 832a8f5 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > e38316b > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_service.py > 704af68 > ambari-server/src/main/resources/common-services/SPARK/1.4.1/kerberos.json > 21ba259 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-env.xml > 5a3edef > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json > 3aafb75 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > c89a302 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > 834871d > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > ea2cf52 > > Diff: https://reviews.apache.org/r/51078/diff/ > > > Testing > --- > > Tested manually. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 51078: AMBARI-18142. Define keytab/principal for Spark Thrift Server
> On Aug. 14, 2016, 3:06 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-env.xml, > > line 76 > > <https://reviews.apache.org/r/51078/diff/1/?file=1472666#file1472666line76> > > > > Can you open a next release JIRA to set all these properties as not to > > be added on ambari upgrade? Its ok for now. > > Jeff Zhang wrote: > Why not adding them on ambari upgrade ? Any concern for that ? > > Sumit Mohanty wrote: > Spark 2.0.0 is added for the first time in Ambari-2.4.0. Where do you see > a scenario when you want these properties get added during Ambari upgrade? If > spark 2.0 exists it will include these properties. If not they will get added > when Spark2 is added. Not sure what the side effect of adding this. I notice all the property of spark2 has this on-ambari-upgrade setting. If this is not correct, then we should remove this for all the properties. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51078/#review145724 ------- On Aug. 14, 2016, 3:45 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51078/ > --- > > (Updated Aug. 14, 2016, 3:45 a.m.) > > > Review request for Ambari and Sumit Mohanty. > > > Bugs: AMBARI-18142 > https://issues.apache.org/jira/browse/AMBARI-18142 > > > Repository: ambari > > > Description > --- > > Should define keytab/principle for spark thrift server in its configuration > file 'spark-thrift-sparkconf' > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-env.xml > 832a8f5 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > e38316b > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_service.py > 704af68 > ambari-server/src/main/resources/common-services/SPARK/1.4.1/kerberos.json > 21ba259 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-env.xml > 5a3edef > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json > 3aafb75 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > c89a302 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > 834871d > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > ea2cf52 > > Diff: https://reviews.apache.org/r/51078/diff/ > > > Testing > --- > > Tested manually. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 51078: AMBARI-18142. Define keytab/principal for Spark Thrift Server
> On Aug. 14, 2016, 3:06 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.4.1/kerberos.json, > > line 38 > > <https://reviews.apache.org/r/51078/diff/1/?file=1472665#file1472665line38> > > > > Should this be supported starting Spark 1.4.1 (HDP-2.4)? Is this also > > an issue when HDP-2.4 is deployed using Ambari-2.2.2? Right, actually customer report this issue in HDP 2.3.4 which use spark 1.4 > On Aug. 14, 2016, 3:06 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-env.xml, > > line 76 > > <https://reviews.apache.org/r/51078/diff/1/?file=1472662#file1472662line76> > > > > Do you want it added during Ambari upgrade for existing clusters? E.g. > > HDP-2.4 deployment with Spark. If not then lets make on-ambari-upgrade to > > false. Yes, since this is a prerty bad bug, I want to fix it when ambari upgrade. > On Aug. 14, 2016, 3:06 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-env.xml, > > line 76 > > <https://reviews.apache.org/r/51078/diff/1/?file=1472666#file1472666line76> > > > > Can you open a next release JIRA to set all these properties as not to > > be added on ambari upgrade? Its ok for now. Why not adding them on ambari upgrade ? Any concern for that ? - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51078/#review145724 --- On Aug. 14, 2016, 3:45 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51078/ > --- > > (Updated Aug. 14, 2016, 3:45 a.m.) > > > Review request for Ambari and Sumit Mohanty. > > > Bugs: AMBARI-18142 > https://issues.apache.org/jira/browse/AMBARI-18142 > > > Repository: ambari > > > Description > --- > > Should define keytab/principle for spark thrift server in its configuration > file 'spark-thrift-sparkconf' > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-env.xml > 832a8f5 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > e38316b > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_service.py > 704af68 > ambari-server/src/main/resources/common-services/SPARK/1.4.1/kerberos.json > 21ba259 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-env.xml > 5a3edef > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json > 3aafb75 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > c89a302 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > 834871d > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > ea2cf52 > > Diff: https://reviews.apache.org/r/51078/diff/ > > > Testing > --- > > Tested manually. > > > Thanks, > > Jeff Zhang > >
Review Request 51078: AMBARI-18142. Define keytab/principal for Spark Thrift Server
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51078/ --- Review request for Ambari and Sumit Mohanty. Bugs: AMBARI-18142 https://issues.apache.org/jira/browse/AMBARI-18142 Repository: ambari Description --- Should define keytab/principle for spark thrift server in its configuration file 'spark-thrift-sparkconf' Diffs - ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-env.xml 832a8f5 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py e38316b ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_service.py 704af68 ambari-server/src/main/resources/common-services/SPARK/1.4.1/kerberos.json 21ba259 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-env.xml 5a3edef ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 3aafb75 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py c89a302 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py 834871d ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json ea2cf52 Diff: https://reviews.apache.org/r/51078/diff/ Testing --- Tested manually. Thanks, Jeff Zhang
Re: Review Request 50973: AMBARI-18072. set JAVA_HOME in livy-env.sh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50973/ --- (Updated Aug. 11, 2016, 11:41 a.m.) Review request for Ambari and Sumit Mohanty. Bugs: AMBARI-18072 https://issues.apache.org/jira/browse/AMBARI-18072 Repository: ambari Description --- Ambari should set JAVA_HOME in livy-env.sh so that user can also start livy manaully. Diffs (updated) - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 6d372e2 ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml c2e6c93 Diff: https://reviews.apache.org/r/50973/diff/ Testing --- Test manullly. Thanks, Jeff Zhang
Review Request 50973: AMBARI-18072. set JAVA_HOME in livy-env.sh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50973/ --- Review request for Ambari and Sumit Mohanty. Bugs: AMBARI-18072 https://issues.apache.org/jira/browse/AMBARI-18072 Repository: ambari Description --- Ambari should set JAVA_HOME in livy-env.sh so that user can also start livy manaully. Diffs - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 6d372e2 ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml c2e6c93 Diff: https://reviews.apache.org/r/50973/diff/ Testing --- Test manullly. Thanks, Jeff Zhang
Review Request 50279: [AMBARI-17827] Spark thrift server should be stopped before RM stop
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50279/ --- Review request for Ambari, Andrew Onischuk and Sumit Mohanty. Bugs: AMBARI-17827 https://issues.apache.org/jira/browse/AMBARI-17827 Repository: ambari Description --- Spark thrift server should be stopped before RM stop, otherwise when RM recovery is enabled, It would try to recover Spark Thrift Server yarn app, this is not what we expect. This issue happens when we kerboerising cluster. Diffs - ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json 624decd ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 9b21ed0 Diff: https://reviews.apache.org/r/50279/diff/ Testing --- Minor configuration chagne, Manually verified. Thanks, Jeff Zhang
Re: Review Request 50231: [AMBARI-17807] Should only upload spark assembly jar once
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50231/ --- (Updated July 20, 2016, 2:47 p.m.) Review request for Ambari, Andrew Onischuk and Sumit Mohanty. Bugs: AMBARI-17807 https://issues.apache.org/jira/browse/AMBARI-17807 Repository: ambari Description --- For now, spark assembly jar will be uploaded multiple times if job history server and thrift server are both installed. It should only been uploaded when job history server is installed. Diffs (updated) - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_service.py c3784d6 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py 0be7270 Diff: https://reviews.apache.org/r/50231/diff/ Testing --- Straightforward change, no test added. Thanks, Jeff Zhang
Review Request 50231: [AMBARI-17807] Should only upload spark assembly jar once
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50231/ --- Review request for Ambari and Sumit Gupta. Bugs: AMBARI-17807 https://issues.apache.org/jira/browse/AMBARI-17807 Repository: ambari Description --- For now, spark assembly jar will be uploaded multiple times if job history server and thrift server are both installed. It should only been uploaded when job history server is installed. Diffs - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_service.py c3784d6 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py 0be7270 Diff: https://reviews.apache.org/r/50231/diff/ Testing --- Straightforward change, no test added. Thanks, Jeff Zhang
Re: Review Request 50185: [AMBARI-17786] Enable csrf protection and add superuser in livy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50185/ --- (Updated July 19, 2016, 12:22 p.m.) Review request for Ambari and Sumit Mohanty. Summary (updated) - [AMBARI-17786] Enable csrf protection and add superuser in livy Bugs: AMBARI-17786 https://issues.apache.org/jira/browse/AMBARI-17786 Repository: ambari Description --- Just a minor configuration change for livy Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml 7d3c78f ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json e479354 Diff: https://reviews.apache.org/r/50185/diff/ Testing --- No test is needed. Just a minor configburation change. Thanks, Jeff Zhang
Re: Review Request 50185: [AMBARI-17786] Enable csrf protection and add superuser
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50185/ --- (Updated July 19, 2016, 12:20 p.m.) Review request for Ambari and Sumit Mohanty. Summary (updated) - [AMBARI-17786] Enable csrf protection and add superuser Bugs: AMBARI-17786 https://issues.apache.org/jira/browse/AMBARI-17786 Repository: ambari Description --- Just a minor configuration change for livy Diffs (updated) - ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml 7d3c78f ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json e479354 Diff: https://reviews.apache.org/r/50185/diff/ Testing (updated) --- No test is needed. Just a minor configburation change. Thanks, Jeff Zhang
Review Request 50185: [AMBARI-17786] Enable csrf protection in livy by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50185/ --- Review request for Ambari and Sumit Mohanty. Bugs: AMBARI-17786 https://issues.apache.org/jira/browse/AMBARI-17786 Repository: ambari Description --- Just a minor configuration change for livy Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml 7d3c78f Diff: https://reviews.apache.org/r/50185/diff/ Testing --- No test is needed. Just a minor configburation chagne. Thanks, Jeff Zhang
Review Request 50116: [AMBARI-17761] Need to specify HADOOP_CONF_DIR in livy-env
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50116/ --- Review request for Ambari and Sumit Mohanty. Bugs: AMBARI-17761 https://issues.apache.org/jira/browse/AMBARI-17761 Repository: ambari Description --- Need to specify HADOOP_CONF_DIR in livy-env Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml 48e96ce Diff: https://reviews.apache.org/r/50116/diff/ Testing --- Tested manully. Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Follow up issue for Spark2 stack definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/ --- (Updated June 22, 2016, 12:06 a.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Changes --- update the unit test Bugs: AMBARI-16920 https://issues.apache.org/jira/browse/AMBARI-16920 Repository: ambari Description --- This a followup ticket for add spark2 stack definition. There's serveral issues: 1. Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml 2. Miss of add spark2 cache file in copy_barball.py 3. Miss the role_commnad_order of spark2 4. conf_select is missign for spark2 Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 4eb0015 ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 286df8d ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py be99edd ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py c2385df ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f6011b0 ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 6c7fe18 Diff: https://reviews.apache.org/r/47941/diff/ Testing --- Manually verified. Thanks, Jeff Zhang
Re: Review Request 48772: [AMBARI-17243] Change livy principal to service type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48772/ --- (Updated June 21, 2016, 2:20 a.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Changes --- add the missing file change. Bugs: AMBARI-17243 https://issues.apache.org/jira/browse/AMBARI-17243 Repository: ambari Description --- 1. livy principal should be service type. 2. Use smoke user to do service check 3. fix livy configuration file name error Diffs (updated) - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py e9ec7d6 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py ba59b95 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py a09a8db ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json d75364b Diff: https://reviews.apache.org/r/48772/diff/ Testing --- Verify it manually. Thanks, Jeff Zhang
Re: Review Request 48772: [AMBARI-17243] Change livy principal to service type
> On June 17, 2016, 1:26 p.m., Robert Levas wrote: > > Ship It! > > Jeff Zhang wrote: > @Robert, Please hold on, I found there's still some issues there. > > Jeff Zhang wrote: > @Robert, please commit it, the issues I found should be due to the messy > enviroment. Sorry, I miss one file, wait a minute. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48772/#review138213 --- On June 17, 2016, 12:58 p.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48772/ > --- > > (Updated June 17, 2016, 12:58 p.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-17243 > https://issues.apache.org/jira/browse/AMBARI-17243 > > > Repository: ambari > > > Description > --- > > 1. livy principal should be service type. > 2. Use smoke user to do service check > 3. fix livy configuration file name error > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py > ba59b95 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py > a09a8db > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > d75364b > > Diff: https://reviews.apache.org/r/48772/diff/ > > > Testing > --- > > Verify it manually. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 48772: [AMBARI-17243] Change livy principal to service type
> On June 17, 2016, 1:26 p.m., Robert Levas wrote: > > Ship It! > > Jeff Zhang wrote: > @Robert, Please hold on, I found there's still some issues there. @Robert, please commit it, the issues I found should be due to the messy enviroment. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48772/#review138213 --- On June 17, 2016, 12:58 p.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48772/ > --- > > (Updated June 17, 2016, 12:58 p.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-17243 > https://issues.apache.org/jira/browse/AMBARI-17243 > > > Repository: ambari > > > Description > --- > > 1. livy principal should be service type. > 2. Use smoke user to do service check > 3. fix livy configuration file name error > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py > ba59b95 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py > a09a8db > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > d75364b > > Diff: https://reviews.apache.org/r/48772/diff/ > > > Testing > --- > > Verify it manually. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 48772: [AMBARI-17243] Change livy principal to service type
> On June 17, 2016, 1:26 p.m., Robert Levas wrote: > > Ship It! @Robert, Please hold on, I found there's still some issues there. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48772/#review138213 --- On June 17, 2016, 12:58 p.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48772/ > --- > > (Updated June 17, 2016, 12:58 p.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-17243 > https://issues.apache.org/jira/browse/AMBARI-17243 > > > Repository: ambari > > > Description > --- > > 1. livy principal should be service type. > 2. Use smoke user to do service check > 3. fix livy configuration file name error > > > Diffs > - > > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py > ba59b95 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py > a09a8db > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > d75364b > > Diff: https://reviews.apache.org/r/48772/diff/ > > > Testing > --- > > Verify it manually. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 48772: [AMBARI-17243] Change livy principal to service type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48772/ --- (Updated June 17, 2016, 12:58 p.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Summary (updated) - [AMBARI-17243] Change livy principal to service type Bugs: AMBARI-17243 https://issues.apache.org/jira/browse/AMBARI-17243 Repository: ambari Description (updated) --- 1. livy principal should be service type. 2. Use smoke user to do service check 3. fix livy configuration file name error Diffs (updated) - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py ba59b95 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py a09a8db ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json d75364b Diff: https://reviews.apache.org/r/48772/diff/ Testing --- Verify it manually. Thanks, Jeff Zhang
Review Request 48848: [AMBARI-17243] Change livy principal to service type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48848/ --- Review request for Ambari, Jayush Luniya and Sumit Mohanty. Bugs: AMBARI-17243 https://issues.apache.org/jira/browse/AMBARI-17243 Repository: ambari Description --- 1. livy principal shoudl be service type. 2. Use smoke user to do service check 3. fix livy configuration file name error Diffs - ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py ba59b95 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py a09a8db ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json d75364b Diff: https://reviews.apache.org/r/48848/diff/ Testing --- Test manually. Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Follow up issue for Spark2 stack definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/ --- (Updated June 16, 2016, 11:06 p.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Changes --- Fix unit test, but not sure how to run. Bugs: AMBARI-16920 https://issues.apache.org/jira/browse/AMBARI-16920 Repository: ambari Description --- This a followup ticket for add spark2 stack definition. There's serveral issues: 1. Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml 2. Miss of add spark2 cache file in copy_barball.py 3. Miss the role_commnad_order of spark2 4. conf_select is missign for spark2 Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 4eb0015 ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 286df8d ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py be99edd ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py c2385df ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f6011b0 ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 6c7fe18 Diff: https://reviews.apache.org/r/47941/diff/ Testing --- Manually verified. Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Follow up issue for Spark2 stack definition
d > > Returning DefaultStackAdvisor implementation > > StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded > > StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded > > Returning XYZ101StackAdvisor implementation > > [INFO] > > > > [INFO] BUILD FAILURE > > [INFO] > > > > [INFO] Total time: 1:15.808s > > [INFO] Finished at: Thu Jun 16 09:56:48 PDT 2016 > > [INFO] Final Memory: 60M/837M > > [INFO] > > I don't have clues why my changes cause unit test failure, could you tell me how to run this specific unit test so that I can reproduce it ? Thanks - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/#review138042 --- On June 15, 2016, 11:06 p.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47941/ > --- > > (Updated June 15, 2016, 11:06 p.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-16920 > https://issues.apache.org/jira/browse/AMBARI-16920 > > > Repository: ambari > > > Description > --- > > This a followup ticket for add spark2 stack definition. There's serveral > issues: > 1. Spark2 thrift server can not started due to miss of > spark-thrift-fairscheduler.xml > 2. Miss of add spark2 cache file in copy_barball.py > 3. Miss the role_commnad_order of spark2 > 4. conf_select is missign for spark2 > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py > 4eb0015 > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py > 286df8d > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > be99edd > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > c2385df > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > f6011b0 > > Diff: https://reviews.apache.org/r/47941/diff/ > > > Testing > --- > > Manually verified. > > > Thanks, > > Jeff Zhang > >
Review Request 48772: [AMBARI-17243] Use " livy-${cluster-name}@${realm}" instead of " livy@${realm}" for identity "livy.server.kerberos.principal"
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48772/ --- Review request for Ambari, Jayush Luniya and Sumit Mohanty. Bugs: AMBARI-17243 https://issues.apache.org/jira/browse/AMBARI-17243 Repository: ambari Description --- To have unique principal names that is the convention followed by rest of the principals. Noticed in stack deploy this principal does not have cluster name. It is a name convention to include cluster name in principals Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json d75364b Diff: https://reviews.apache.org/r/48772/diff/ Testing --- Verify it manually. Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Follow up issue for Spark2 stack definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/ --- (Updated June 15, 2016, 11:06 p.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Changes --- Patch rebased Bugs: AMBARI-16920 https://issues.apache.org/jira/browse/AMBARI-16920 Repository: ambari Description --- This a followup ticket for add spark2 stack definition. There's serveral issues: 1. Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml 2. Miss of add spark2 cache file in copy_barball.py 3. Miss the role_commnad_order of spark2 4. conf_select is missign for spark2 Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 4eb0015 ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 286df8d ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py be99edd ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py c2385df ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f6011b0 Diff: https://reviews.apache.org/r/47941/diff/ Testing --- Manually verified. Thanks, Jeff Zhang
Re: Review Request 48308: [AMBARI-17078] Make Spark2-ThriftServer and Livy Server as optional by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48308/ --- (Updated June 14, 2016, 4:10 a.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Changes --- Move the change to HDP 2.5 Bugs: AMBARI-17078 https://issues.apache.org/jira/browse/AMBARI-17078 Repository: ambari Description --- Spark2-ThriftServer and Livy Server are not being selected by default, but should be unselected by default. Diffs (updated) - ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py db2986b Diff: https://reviews.apache.org/r/48308/diff/ Testing --- Manullay verified Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Follow up issue for Spark2 stack definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/ --- (Updated June 13, 2016, 10:45 a.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Bugs: AMBARI-16920 https://issues.apache.org/jira/browse/AMBARI-16920 Repository: ambari Description (updated) --- This a followup ticket for add spark2 stack definition. There's serveral issues: 1. Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml 2. Miss of add spark2 cache file in copy_barball.py 3. Miss the role_commnad_order of spark2 4. conf_select is missign for spark2 Diffs - ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 4eb0015 ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 286df8d ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py 6925ab5 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py c2385df ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f6011b0 Diff: https://reviews.apache.org/r/47941/diff/ Testing --- Manually verified. Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Follow up issue for Spark2 stack definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/ --- (Updated June 13, 2016, 10:28 a.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Changes --- address comments and rebase patch Summary (updated) - [AMBARI-16920] Follow up issue for Spark2 stack definition Bugs: AMBARI-16920 https://issues.apache.org/jira/browse/AMBARI-16920 Repository: ambari Description --- This a followup ticket for add spark2 stack definition. There's serveral issues: 1. Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml 2. Miss of add spark2 cache file in copy_barball.py 3. Miss the role_commnad_order of spark2 Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 4eb0015 ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 286df8d ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py 6925ab5 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py c2385df ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f6011b0 Diff: https://reviews.apache.org/r/47941/diff/ Testing --- Manually verified. Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/ --- (Updated June 8, 2016, 7:31 a.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Changes --- Update patch to address comments. Bugs: AMBARI-16920 https://issues.apache.org/jira/browse/AMBARI-16920 Repository: ambari Description --- This a followup ticket for add spark2 stack definition. There's serveral issues: 1. Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml 2. Miss of add spark2 cache file in copy_barball.py 3. Miss the role_commnad_order of spark2 Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 286df8d ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py 3316f78 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py 2eae3e7 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 1b33345 Diff: https://reviews.apache.org/r/47941/diff/ Testing --- Manually verified. Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml
> On June 8, 2016, 1:08 a.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py, > > line 61 > > <https://reviews.apache.org/r/47941/diff/2/?file=1408197#file1408197line61> > > > > This is calling what's meant to be a private function and getting the > > element at index 1 without checking the length. > > > > Rename the function by removing the leading "_" and check the length > > before accessing [1] This method return a tuple, so I think it is not necessary to check the length. If the length is incorrect, then must be something wrong, the installation should not continue. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/#review136568 --- On June 7, 2016, 6:16 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47941/ > --- > > (Updated June 7, 2016, 6:16 a.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-16920 > https://issues.apache.org/jira/browse/AMBARI-16920 > > > Repository: ambari > > > Description > --- > > This a followup ticket for add spark2 stack definition. There's serveral > issues: > 1. Spark2 thrift server can not started due to miss of > spark-thrift-fairscheduler.xml > 2. Miss of add spark2 cache file in copy_barball.py > 3. Miss the role_commnad_order of spark2 > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py > 286df8d > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > 3316f78 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > 2eae3e7 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > 1b33345 > > Diff: https://reviews.apache.org/r/47941/diff/ > > > Testing > --- > > Manually verified. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml
> On June 8, 2016, 1:08 a.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py, > > line 63 > > <https://reviews.apache.org/r/47941/diff/2/?file=1408197#file1408197line63> > > > > Why replace_existing_files=True? > > This will make it slower on Start You have one comment about "This tarball should be deleted and recreated every time the service is restarted.", does this mean to upload new tarball when service is restarted ? - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/#review136568 ------- On June 7, 2016, 6:16 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47941/ > --- > > (Updated June 7, 2016, 6:16 a.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-16920 > https://issues.apache.org/jira/browse/AMBARI-16920 > > > Repository: ambari > > > Description > --- > > This a followup ticket for add spark2 stack definition. There's serveral > issues: > 1. Spark2 thrift server can not started due to miss of > spark-thrift-fairscheduler.xml > 2. Miss of add spark2 cache file in copy_barball.py > 3. Miss the role_commnad_order of spark2 > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py > 286df8d > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > 3316f78 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > 2eae3e7 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > 1b33345 > > Diff: https://reviews.apache.org/r/47941/diff/ > > > Testing > --- > > Manually verified. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/ --- (Updated June 7, 2016, 6:16 a.m.) Review request for Ambari, Jayush Luniya and Sumit Mohanty. Changes --- Address the comments Bugs: AMBARI-16920 https://issues.apache.org/jira/browse/AMBARI-16920 Repository: ambari Description --- This a followup ticket for add spark2 stack definition. There's serveral issues: 1. Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml 2. Miss of add spark2 cache file in copy_barball.py 3. Miss the role_commnad_order of spark2 Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 286df8d ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py 3316f78 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py 2eae3e7 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 1b33345 Diff: https://reviews.apache.org/r/47941/diff/ Testing --- Manually verified. Thanks, Jeff Zhang
Review Request 48308: [AMBARI-17078] Make Spark2-ThriftServer and Livy Server as optional by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48308/ --- Review request for Ambari, Jayush Luniya and Sumit Mohanty. Bugs: AMBARI-17078 https://issues.apache.org/jira/browse/AMBARI-17078 Repository: ambari Description --- Spark2-ThriftServer and Livy Server are not being selected by default, but should be unselected by default. Diffs - ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 36fe066b65b12326a128fdb5e8caf9ad56a8211c Diff: https://reviews.apache.org/r/48308/diff/ Testing --- Manullay verified Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml
> On June 3, 2016, 12:01 a.m., Alejandro Fernandez wrote: > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py, > > line 63 > > <https://reviews.apache.org/r/47941/diff/1/?file=1396056#file1396056line63> > > > > configure() can be called on refresh configs, while the service is > > running. What about delete on stop command. A restart will issue a stop > > followed by a start. > > If a start command fails, a try-catch-finally can do some cleanup and > > delete the tarball. > > Start command should be a no-op if the service is already running Just notice copy_to_hdfs has one parameter to allow replace the existing tarball, I think I use this api. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/#review136015 --- On May 27, 2016, 3:33 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47941/ > --- > > (Updated May 27, 2016, 3:33 a.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-16920 > https://issues.apache.org/jira/browse/AMBARI-16920 > > > Repository: ambari > > > Description > --- > > This a followup ticket for add spark2 stack definition. There's serveral > issues: > 1. Spark2 thrift server can not started due to miss of > spark-thrift-fairscheduler.xml > 2. Miss of add spark2 cache file in copy_barball.py > 3. Miss the role_commnad_order of spark2 > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py > 286df8d > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > ded9959 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > 2eae3e7 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > 4a7c1d2 > > Diff: https://reviews.apache.org/r/47941/diff/ > > > Testing > --- > > Manually verified. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml
> On June 2, 2016, 4:59 p.m., Alejandro Fernandez wrote: > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py, > > line 63 > > <https://reviews.apache.org/r/47941/diff/1/?file=1396056#file1396056line63> > > > > This tarball should be deleted and recreated every time the service is > > restarted. Do you mean the tarball on hdfs should be deleted ? I have one concern that if one spark application is running when the service is started (it is possible), then it is possile to fail the spark application due to missing of this tarball. So I think it would be better to just create and upload this tar only when service is configured. I notice hive will upload tarball only in service config phase, we might need to that the same way, thoughts ? - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/#review135965 --- On May 27, 2016, 3:33 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47941/ > --- > > (Updated May 27, 2016, 3:33 a.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-16920 > https://issues.apache.org/jira/browse/AMBARI-16920 > > > Repository: ambari > > > Description > --- > > This a followup ticket for add spark2 stack definition. There's serveral > issues: > 1. Spark2 thrift server can not started due to miss of > spark-thrift-fairscheduler.xml > 2. Miss of add spark2 cache file in copy_barball.py > 3. Miss the role_commnad_order of spark2 > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py > 286df8d > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > ded9959 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > 2eae3e7 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > 4a7c1d2 > > Diff: https://reviews.apache.org/r/47941/diff/ > > > Testing > --- > > Manually verified. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated June 2, 2016, 10:27 a.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- upload addendum patch Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 46c7b9d Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated June 1, 2016, 11:06 p.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py eba1abf ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py 9a3fa31 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 3f14955 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 99a3ade ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py d2e0d11 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 4a7c1d2 ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-spark-blacklist.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated May 31, 2016, 12:53 p.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- rebase patch Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py eba1abf ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py 9a3fa31 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 3f14955 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 99a3ade ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py d2e0d11 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 4a7c1d2 ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-spark-blacklist.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated May 31, 2016, 12:52 p.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- rebase patch Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py eba1abf ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py 9a3fa31 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 3f14955 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 99a3ade ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py d2e0d11 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 4a7c1d2 ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-spark-blacklist.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml
> On May 27, 2016, 5:48 p.m., Alejandro Fernandez wrote: > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py, > > line 63 > > <https://reviews.apache.org/r/47941/diff/1/?file=1396056#file1396056line63> > > > > Why is the tarball source coming from /tmp? We creat tarball in ambari at installtion instead of installing from rpm. Because we want to have full control of what do be included in tarball rather than depdends on rpm. please refer ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/#review135273 ------- On May 27, 2016, 3:33 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47941/ > --- > > (Updated May 27, 2016, 3:33 a.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-16920 > https://issues.apache.org/jira/browse/AMBARI-16920 > > > Repository: ambari > > > Description > --- > > This a followup ticket for add spark2 stack definition. There's serveral > issues: > 1. Spark2 thrift server can not started due to miss of > spark-thrift-fairscheduler.xml > 2. Miss of add spark2 cache file in copy_barball.py > 3. Miss the role_commnad_order of spark2 > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py > 286df8d > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py > ded9959 > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py > 2eae3e7 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > 4a7c1d2 > > Diff: https://reviews.apache.org/r/47941/diff/ > > > Testing > --- > > Manually verified. > > > Thanks, > > Jeff Zhang > >
Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47941/ --- Review request for Ambari, Jayush Luniya and Sumit Mohanty. Bugs: AMBARI-16920 https://issues.apache.org/jira/browse/AMBARI-16920 Repository: ambari Description --- This a followup ticket for add spark2 stack definition. There's serveral issues: 1. Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml 2. Miss of add spark2 cache file in copy_barball.py 3. Miss the role_commnad_order of spark2 Diffs - ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 286df8d ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py ded9959 ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py 2eae3e7 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 4a7c1d2 Diff: https://reviews.apache.org/r/47941/diff/ Testing --- Manually verified. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
> On May 18, 2016, 11:53 p.m., Jayush Luniya wrote: > > @Jeff > > Can you verify that blueprint deployment of the cluster without Livy Server > > works? > > > > Guide on Blueprints > > https://cwiki.apache.org/confluence/display/AMBARI/Blueprints > > Jayush Luniya wrote: > @Jeff whats the latest on this? Did you get the blueprint scenario > verified? Also can you update the patch with latest in trunk? > > Jeff Zhang wrote: > @Jayush, I have some issues when verifying blueprint, also ask for help > to ambari mail list, but still not progress yet, could you help on that ? > http://mail-archives.apache.org/mod_mbox/ambari-user/201605.mbox/%3CCAADy7x7ha6amOjmNFCQN2bKpMACGbLJr3uDxKjPCn=jph_7...@mail.gmail.com%3E Ping @Jayush - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/#review133841 --- On May 26, 2016, 3:22 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46727/ > --- > > (Updated May 26, 2016, 3:22 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit > Mohanty. > > > Bugs: AMBARI-15612 > https://issues.apache.org/jira/browse/AMBARI-15612 > > > Repository: ambari > > > Description > --- > > Add Livy to HDP 2.5 as slave component of Spark. > > Move the spark stack definiation to commoon-services and then > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > 775e7eb > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > aa64528 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > c5f3eb6 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py > 694f046 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py > 86e7f7d > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > 3cae192 > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 6e506a0 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > 4013835 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-spark-blacklist.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml > d1129cd > > Diff: https://reviews.apache.org/r/46727/diff/ > > > Testing > --- > > Manully verify it. Livy is installed correctly in non-secure and secure > cluster. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated May 26, 2016, 3:22 a.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- Rebase patch Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py 775e7eb ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py aa64528 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py c5f3eb6 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 3cae192 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 6e506a0 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 4013835 ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-spark-blacklist.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
> On May 18, 2016, 11:53 p.m., Jayush Luniya wrote: > > @Jeff > > Can you verify that blueprint deployment of the cluster without Livy Server > > works? > > > > Guide on Blueprints > > https://cwiki.apache.org/confluence/display/AMBARI/Blueprints > > Jayush Luniya wrote: > @Jeff whats the latest on this? Did you get the blueprint scenario > verified? Also can you update the patch with latest in trunk? @Jayush, I have some issues when verifying blueprint, also ask for help to ambari mail list, but still not progress yet, could you help on that ? http://mail-archives.apache.org/mod_mbox/ambari-user/201605.mbox/%3CCAADy7x7ha6amOjmNFCQN2bKpMACGbLJr3uDxKjPCn=jph_7...@mail.gmail.com%3E - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/#review133841 --- On May 18, 2016, 11:10 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46727/ > --- > > (Updated May 18, 2016, 11:10 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit > Mohanty. > > > Bugs: AMBARI-15612 > https://issues.apache.org/jira/browse/AMBARI-15612 > > > Repository: ambari > > > Description > --- > > Add Livy to HDP 2.5 as slave component of Spark. > > Move the spark stack definiation to commoon-services and then > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > 2e18443 > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > 9e9cadc > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > c5f3eb6 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py > 694f046 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py > 86e7f7d > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > 3ec5d05 > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 6e506a0 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > f7e15aa > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-spark-blacklist.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml > d1129cd > > Diff: https://reviews.apache.org/r/46727/diff/ > > > Testing > --- > > Manully verify it. Livy is installed correctly in non-secure and secure > cluster. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
+ Sean Roberts who may able to help
Hi Jayush & Sean,
I also ask the question on ambari mail list, but still could not make progress,
not sure how to verify that, do you have any suggestion on that ? Very
appreciated.
http://mail-archives.apache.org/mod_mbox/ambari-user/201605.mbox/%3CCAADy7x7ha6amOjmNFCQN2bKpMACGbLJr3uDxKjPCn=jph_7...@mail.gmail.com%3E
Best Regard,
Jeff Zhang
From: Jianfeng Zhang <jzh...@hortonworks.com<mailto:jzh...@hortonworks.com>>
Date: Friday, May 20, 2016 at 7:28 AM
To: Jayush Luniya <jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>,
Sumit Mohanty <smoha...@hortonworks.com<mailto:smoha...@hortonworks.com>>,
Alejandro Fernandez
<afernan...@hortonworks.com<mailto:afernan...@hortonworks.com>>
Cc: Jeff Zhang <zjf...@apache.org<mailto:zjf...@apache.org>>, Ambari
<reviews@ambari.apache.org<mailto:reviews@ambari.apache.org>>
Subject: Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave
component of Spark
Hi Jayush，
Could you help take a look at it ? Thanks.
Best Regard,
Jeff Zhang
From: Jianfeng Zhang <jzh...@hortonworks.com<mailto:jzh...@hortonworks.com>>
Date: Thursday, May 19, 2016 at 10:22 AM
To: Jayush Luniya <jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>,
Sumit Mohanty <smoha...@hortonworks.com<mailto:smoha...@hortonworks.com>>,
Alejandro Fernandez
<afernan...@hortonworks.com<mailto:afernan...@hortonworks.com>>
Cc: Jeff Zhang <zjf...@apache.org<mailto:zjf...@apache.org>>, Ambari
<reviews@ambari.apache.org<mailto:reviews@ambari.apache.org>>
Subject: Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave
component of Spark
Hi Jayush,
I try to verify it on blueprints, but fails with 2 problems.
One is when setup hw repo, another is when registering blueprint. Here’s the
errors I get, could you guide me what steps I miss here ? Thanks
Setup hw repo.
curl -X PUT -u admin:admin -d @repo.json -H 'X-Requested-By: ambari'
http://jzhang-test3-1.openstacklocal:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/centos6/repositories/hw_repo
Error I get:
{
"status" : 404,
"message" : "org.apache.ambari.server.controller.spi.NoSuchResourceException:
The specified resource doesn't exist: Stack data, stackName=HDP,
stackVersion=2.5, osType=centos6, repoId=hw_repo”
}
Register blueprint
curl -X POST -u admin:admin -d @dev_cluster.json -H 'X-Requested-By: ambari'
http://jzhang-test3-1.openstacklocal:8080/api/v1/blueprints/dev_cluster
Error I get
{
"status" : 400,
"message" : "Blueprint configuration validation failed: Missing required
properties. Specify a value for these properties in the blueprint
configuration. {host_group_1={kerberos-env=[kdc_type, kdc_hosts,
encryption_types, realm, ldap_url, container_dn]}}”
}
Attach are the request body file for these 2 cases.
Best Regard,
Jeff Zhang
From: Jayush Luniya
<nore...@reviews.apache.org<mailto:nore...@reviews.apache.org>> on behalf of
Jayush Luniya <jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>
Reply-To: Jayush Luniya
<jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>
Date: Thursday, May 19, 2016 at 7:53 AM
To: Jayush Luniya <jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>,
Sumit Mohanty <smoha...@hortonworks.com<mailto:smoha...@hortonworks.com>>,
Alejandro Fernandez
<afernan...@hortonworks.com<mailto:afernan...@hortonworks.com>>
Cc: Jeff Zhang <zjf...@apache.org<mailto:zjf...@apache.org>>, Ambari
<reviews@ambari.apache.org<mailto:reviews@ambari.apache.org>>
Subject: Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave
component of Spark
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46727/
@Jeff
Can you verify that blueprint deployment of the cluster without Livy Server
works?
Guide on Blueprints
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
- Jayush Luniya
On May 18th, 2016, 11:10 a.m. UTC, Jeff Zhang wrote:
Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit
Mohanty.
By Jeff Zhang.
Updated May 18, 2016, 11:10 a.m.
Bugs: AMBARI-15612<https://issues.apache.org/jira/browse/AMBARI-15612>
Repository: ambari
Description
Add Livy to HDP 2.5 as slave component of Spark.
Move the spark stack definiation to commoon-services and then
Testing
Manully verify it. Livy is installed correctly in non-secure and secure cluster.
Diffs
*
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
(2e18443)
*
ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
(9e9cadc)
*
ambari-server/src/main/resources/common-se
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
Hi Jayush，
Could you help take a look at it ? Thanks.
Best Regard,
Jeff Zhang
From: Jianfeng Zhang <jzh...@hortonworks.com<mailto:jzh...@hortonworks.com>>
Date: Thursday, May 19, 2016 at 10:22 AM
To: Jayush Luniya <jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>,
Sumit Mohanty <smoha...@hortonworks.com<mailto:smoha...@hortonworks.com>>,
Alejandro Fernandez
<afernan...@hortonworks.com<mailto:afernan...@hortonworks.com>>
Cc: Jeff Zhang <zjf...@apache.org<mailto:zjf...@apache.org>>, Ambari
<reviews@ambari.apache.org<mailto:reviews@ambari.apache.org>>
Subject: Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave
component of Spark
Hi Jayush,
I try to verify it on blueprints, but fails with 2 problems.
One is when setup hw repo, another is when registering blueprint. Here’s the
errors I get, could you guide me what steps I miss here ? Thanks
Setup hw repo.
curl -X PUT -u admin:admin -d @repo.json -H 'X-Requested-By: ambari'
http://jzhang-test3-1.openstacklocal:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/centos6/repositories/hw_repo
Error I get:
{
"status" : 404,
"message" : "org.apache.ambari.server.controller.spi.NoSuchResourceException:
The specified resource doesn't exist: Stack data, stackName=HDP,
stackVersion=2.5, osType=centos6, repoId=hw_repo”
}
Register blueprint
curl -X POST -u admin:admin -d @dev_cluster.json -H 'X-Requested-By: ambari'
http://jzhang-test3-1.openstacklocal:8080/api/v1/blueprints/dev_cluster
Error I get
{
"status" : 400,
"message" : "Blueprint configuration validation failed: Missing required
properties. Specify a value for these properties in the blueprint
configuration. {host_group_1={kerberos-env=[kdc_type, kdc_hosts,
encryption_types, realm, ldap_url, container_dn]}}”
}
Attach are the request body file for these 2 cases.
Best Regard,
Jeff Zhang
From: Jayush Luniya
<nore...@reviews.apache.org<mailto:nore...@reviews.apache.org>> on behalf of
Jayush Luniya <jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>
Reply-To: Jayush Luniya
<jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>
Date: Thursday, May 19, 2016 at 7:53 AM
To: Jayush Luniya <jlun...@hortonworks.com<mailto:jlun...@hortonworks.com>>,
Sumit Mohanty <smoha...@hortonworks.com<mailto:smoha...@hortonworks.com>>,
Alejandro Fernandez
<afernan...@hortonworks.com<mailto:afernan...@hortonworks.com>>
Cc: Jeff Zhang <zjf...@apache.org<mailto:zjf...@apache.org>>, Ambari
<reviews@ambari.apache.org<mailto:reviews@ambari.apache.org>>
Subject: Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave
component of Spark
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46727/
@Jeff
Can you verify that blueprint deployment of the cluster without Livy Server
works?
Guide on Blueprints
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
- Jayush Luniya
On May 18th, 2016, 11:10 a.m. UTC, Jeff Zhang wrote:
Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit
Mohanty.
By Jeff Zhang.
Updated May 18, 2016, 11:10 a.m.
Bugs: AMBARI-15612<https://issues.apache.org/jira/browse/AMBARI-15612>
Repository: ambari
Description
Add Livy to HDP 2.5 as slave component of Spark.
Move the spark stack definiation to commoon-services and then
Testing
Manully verify it. Livy is installed correctly in non-secure and secure cluster.
Diffs
*
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
(2e18443)
*
ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
(9e9cadc)
*
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py
(PRE-CREATION)
*
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
(PRE-CREATION)
*
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
(c5f3eb6)
*
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py
(694f046)
*
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py
(PRE-CREATION)
*
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py
(86e7f7d)
*
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
(3ec5d05)
*
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py
(6e506a0)
* ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json
(f7e15aa)
*
ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-conf.xml
(PRE-CREATION)
*
ambari-server/src/main/resou
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated May 17, 2016, 12:17 a.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- Remove mis-uploaded file Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py 2e18443 ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py 9e9cadc ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py c5f3eb6 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 3ec5d05 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 6e506a0 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f7e15aa ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-defaults.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-spark-blacklist.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated May 17, 2016, 12:11 a.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- Change the cardinality of livy and add new conf file spark-blacklist.proerties. Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py 2e18443 ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py 9e9cadc ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py c5f3eb6 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 3ec5d05 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 6e506a0 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f7e15aa ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-defaults.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-spark-blacklist.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd ambari-server/src/test/python/stacks/2.5/SPARK/test_livy_server.py PRE-CREATION Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated May 14, 2016, 12:56 a.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- rebase patch Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py 2e18443 ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py 9e9cadc ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py c5f3eb6 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 3ec5d05 ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 6e506a0 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f7e15aa ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-defaults.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated May 12, 2016, 12:41 a.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- minor update on role_command_order.json Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py abd2212 ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py daa97de ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py c5f3eb6 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json aabc37d ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 6e506a0 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f7e15aa ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-defaults.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
> On May 7, 2016, 7:31 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py, > > line 280 > > <https://reviews.apache.org/r/46727/diff/7/?file=1375688#file1375688line280> > > > > We should add this to HDP25StackAdvisor instead? > > > > > > ambari/ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py The method recommendHadoopProxyUsers is for hadoop configuration. I feel it is better to put it here. Otherwise I have to copy part of code in recommendHadoopProxyUsers to HDP25StackAdvisor.py which is not good for code reuse and maintenance. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/#review132146 --- On May 7, 2016, 3:09 p.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46727/ > --- > > (Updated May 7, 2016, 3:09 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit > Mohanty. > > > Bugs: AMBARI-15612 > https://issues.apache.org/jira/browse/AMBARI-15612 > > > Repository: ambari > > > Description > --- > > Add Livy to HDP 2.5 as slave component of Spark. > > Move the spark stack definiation to commoon-services and then > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > abd2212 > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > daa97de > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > c5f3eb6 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py > 694f046 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py > 86e7f7d > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > aabc37d > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 6e506a0 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > f7e15aa > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml > d1129cd > > Diff: https://reviews.apache.org/r/46727/diff/ > > > Testing > --- > > Manully verify it. Livy is installed correctly in non-secure and secure > cluster. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated May 7, 2016, 3:09 p.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit Mohanty. Changes --- Rebase the patch, but unfornatelly it still fails for the same issue of previous patch. check_stack_feature(StackFeature.SPARK_LIVY, stack_version_formatted) always return False. Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py abd2212 ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py daa97de ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py c5f3eb6 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json aabc37d ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 6e506a0 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f7e15aa ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-defaults.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml d1129cd Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
> On April 29, 2016, 12:25 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py, > > line 45 > > <https://reviews.apache.org/r/46727/diff/4/?file=1365338#file1365338line45> > > > > Will Spark hist server and livy server always allow http calls? Do we > > need support for https? > > > > I also think that we can reduce the retries to 5 times. Any reason > > service_check should keep on trying 10 times? https is supported. But user have to enable it manually because ambari seems don't have the ability to generate keystore file. Will change the retries to 5 times. > On April 29, 2016, 12:25 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-defaults.xml, > > line 44 > > <https://reviews.apache.org/r/46727/diff/4/?file=1365345#file1365345line44> > > > > One hour seems too long. Will it have any impact of start/service check? It is default setting of livy. It is session timeout, should not related with service check. > On April 29, 2016, 12:25 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml, > > line 27 > > <https://reviews.apache.org/r/46727/diff/4/?file=1365353#file1365353line27> > > > > What is the motivation to move these to common-services? This is for easy code manintance. Considering spark evolve much faster than other components, so I put all of the spark code under common-service, the new version extend the old version. All the python script is in the oldest one 1.2.0.2.2. In the real HDP stack definition, it just either extend from common-service or extend from its parent HDP stack. > On April 29, 2016, 12:25 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json, > > line 10 > > <https://reviews.apache.org/r/46727/diff/4/?file=1365354#file1365354line10> > > > > Why does LIVY server need all the four to be started? > > > > Does SPARK service check also need to wait for LIVY server start? hmm. Seems livy service start doesn't need these depedency. And yes spark service check need to wait for livy server start, because spark service check include livy service check. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/#review131041 --- On April 29, 2016, 7:45 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46727/ > --- > > (Updated April 29, 2016, 7:45 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit > Mohanty. > > > Bugs: AMBARI-15612 > https://issues.apache.org/jira/browse/AMBARI-15612 > > > Repository: ambari > > > Description > --- > > Add Livy to HDP 2.5 as slave component of Spark. > > Move the spark stack definiation to commoon-services and then > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > abd2212 > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > daa97de > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_server.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_service.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py > c5f3eb6 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py > 694f046 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_livy.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py > 86e7f7d > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-fairscheduler.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-sparkconf.xml > PRE-CREATION > > ambari-server/src/main/resources/commo
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
> On April 29, 2016, 10:10 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_server.py, > > line 61 > > <https://reviews.apache.org/r/46727/diff/5/?file=1365607#file1365607line61> > > > > get_stack_to_compoent is now been removed and replaced with > > get_component_name(). See AMBARI-15609 Will fix it - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/#review131201 ------- On April 29, 2016, 7:45 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46727/ > --- > > (Updated April 29, 2016, 7:45 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit > Mohanty. > > > Bugs: AMBARI-15612 > https://issues.apache.org/jira/browse/AMBARI-15612 > > > Repository: ambari > > > Description > --- > > Add Livy to HDP 2.5 as slave component of Spark. > > Move the spark stack definiation to commoon-services and then > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > abd2212 > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > daa97de > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_server.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_service.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py > c5f3eb6 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py > 694f046 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_livy.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py > 86e7f7d > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-fairscheduler.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-sparkconf.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/metainfo.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-env.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-log4j-properties.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/kerberos.json > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/metainfo.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml > 1a6552f > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml > 2dda4bb > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml > 4f2563b > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml > a5a19f6 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > f7e15aa > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml > 107ca93 > > Diff: https://reviews.apache.org/r/46727/diff/ > > > Testing > --- > > Manully verify it. Livy is installed correctly in non-secure and secure > cluster. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
> On April 29, 2016, 9:53 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/metainfo.xml, > > line 1 > > <https://reviews.apache.org/r/46727/diff/5/?file=1365616#file1365616line1> > > > > We shouldnt name this as 1.6.x.2.4. Just SPARK/1.6.0. I dont know if a > > non-digit charater here will cause any issues. The last 2-digits "2.4" is > > really HDP version and should not be added. Yes we have 5-digit numbering > > in common-services with HDP version in it but it is incorrect and needs to > > be fixed :). In the HDP stack we can override the version in > > stacks/HDP/2.4/services/SPARK/metainfo.xml if we want to be specific as say > > "1.6.1.2.4" The non-digit is added in https://issues.apache.org/jira/browse/AMBARI-15497 , I think the purpose is for avoid changing ambari code each time when spark version changes because spark evolution very rapidly. And I notice other service under commonservices are all with HDP version at the end. Does it really need to be removed ? - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/#review131197 --- On April 29, 2016, 7:45 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46727/ > --- > > (Updated April 29, 2016, 7:45 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit > Mohanty. > > > Bugs: AMBARI-15612 > https://issues.apache.org/jira/browse/AMBARI-15612 > > > Repository: ambari > > > Description > --- > > Add Livy to HDP 2.5 as slave component of Spark. > > Move the spark stack definiation to commoon-services and then > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > abd2212 > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > daa97de > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_server.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_service.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py > c5f3eb6 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py > 694f046 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_livy.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py > 86e7f7d > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-fairscheduler.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-sparkconf.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/metainfo.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-env.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-log4j-properties.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/kerberos.json > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/metainfo.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml > 1a6552f > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml > 2dda4bb > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml > 4f2563b > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml > a5a19f6 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > f7e15aa > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml > 107ca93 > > Diff: https://reviews.apache.org/r/46727/diff/ > > > Testing > --- > > Manully verify it. Livy is installed correctly in non-secure and secure > cluster. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
> On April 29, 2016, 10:07 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/kerberos.json, > > line 1 > > <https://reviews.apache.org/r/46727/diff/5/?file=1365620#file1365620line1> > > > > Having 1.6.x.2.4 and 1.6.x.2.5 is definitely confusing especially for > > Non-HDP stack. We really need to make common-services truely HDP agnostic. As I mentioned above, I think we should create a new ticket for removing HDP version for all the services under common-services. > On April 29, 2016, 10:07 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml, > > line 26 > > <https://reviews.apache.org/r/46727/diff/5/?file=1365625#file1365625line26> > > > > Why do we call this 1.6.x.2.4? In the UI we will display version as > > 1.6.x.2.4 as well I think it is OK, this is purpose of AMBARI-15497 - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/#review131200 ------- On April 29, 2016, 7:45 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46727/ > --- > > (Updated April 29, 2016, 7:45 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit > Mohanty. > > > Bugs: AMBARI-15612 > https://issues.apache.org/jira/browse/AMBARI-15612 > > > Repository: ambari > > > Description > --- > > Add Livy to HDP 2.5 as slave component of Spark. > > Move the spark stack definiation to commoon-services and then > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > abd2212 > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > daa97de > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_server.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_service.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py > c5f3eb6 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py > 694f046 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_livy.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py > 86e7f7d > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-fairscheduler.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-sparkconf.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/metainfo.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-env.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-log4j-properties.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/kerberos.json > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/metainfo.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml > 1a6552f > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml > 2dda4bb > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml > 4f2563b > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml > a5a19f6 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > f7e15aa > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml > 107ca93 > > Diff: https://reviews.apache.org/r/46727/diff/ > > > Testing > --- > > Manully verify it. Livy is installed correctly in non-secure and secure > cluster. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
> On April 29, 2016, 9:57 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-fairscheduler.xml, > > line 24 > > <https://reviews.apache.org/r/46727/diff/5/?file=1365614#file1365614line24> > > > > We now support reading properties from files. You can put this in > > common-services/SPARK/1.6.1/properties/fairscheduler.xml and read from > > there. That way you dont have to embed an encoded xml string within the > > config xml. See https://issues.apache.org/jira/browse/AMBARI-15842 as an > > example. For this kind of code refactoring, I'd like to do it in another ticket in case it would introduce new bugs. - Jeff --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/#review131199 --- On April 29, 2016, 7:45 a.m., Jeff Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46727/ > --- > > (Updated April 29, 2016, 7:45 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit > Mohanty. > > > Bugs: AMBARI-15612 > https://issues.apache.org/jira/browse/AMBARI-15612 > > > Repository: ambari > > > Description > --- > > Add Livy to HDP 2.5 as slave component of Spark. > > Move the spark stack definiation to commoon-services and then > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > abd2212 > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > daa97de > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_server.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_service.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py > c5f3eb6 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py > 694f046 > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_livy.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py > 86e7f7d > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-fairscheduler.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-sparkconf.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/metainfo.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-env.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-log4j-properties.xml > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/kerberos.json > PRE-CREATION > > ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/metainfo.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml > 1a6552f > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml > 2dda4bb > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml > 4f2563b > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml > a5a19f6 > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > f7e15aa > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml > 107ca93 > > Diff: https://reviews.apache.org/r/46727/diff/ > > > Testing > --- > > Manully verify it. Livy is installed correctly in non-secure and secure > cluster. > > > Thanks, > > Jeff Zhang > >
Re: Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- (Updated April 29, 2016, 3 a.m.) Review request for Ambari and Sumit Mohanty. Changes --- add LIVY_SERVER_JAVA_OPTS to livy-env.sh Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark. Move the spark stack definiation to commoon-services and then Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py daa97de ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py c5f3eb6 ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py 86e7f7d ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-defaults.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-fairscheduler.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-sparkconf.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/metainfo.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-defaults.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/kerberos.json PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/metainfo.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml 1a6552f ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml 2dda4bb ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml 4f2563b ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml a5a19f6 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f7e15aa ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml 107ca93 Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
Review Request 46727: [AMBARI-15612] Add Livy to HDP 2.5 as slave component of Spark
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46727/ --- Review request for Ambari and Sumit Mohanty. Bugs: AMBARI-15612 https://issues.apache.org/jira/browse/AMBARI-15612 Repository: ambari Description --- Add Livy to HDP 2.5 as slave component of Spark Diffs - ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_server.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/livy_service.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py a64f36f ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py 694f046 ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_livy.py PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py 19c9a0e ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-defaults.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-fairscheduler.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/configuration/spark-thrift-sparkconf.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.4/metainfo.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-defaults.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-env.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/configuration/livy-log4j-properties.xml PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/kerberos.json PRE-CREATION ambari-server/src/main/resources/common-services/SPARK/1.6.x.2.5/metainfo.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml 1a6552f ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml 2dda4bb ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml 4f2563b ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml a5a19f6 ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json f7e15aa ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/metainfo.xml 107ca93 Diff: https://reviews.apache.org/r/46727/diff/ Testing --- Manully verify it. Livy is installed correctly in non-secure and secure cluster. Thanks, Jeff Zhang
