[jira] [Created] (SLIDER-650) regression: local zk nodes not being deleted on instance destroy
Steve Loughran created SLIDER-650: - Summary: regression: local zk nodes not being deleted on instance destroy Key: SLIDER-650 URL: https://issues.apache.org/jira/browse/SLIDER-650 Project: Slider Issue Type: Bug Components: client Affects Versions: Slider 0.60 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Blocker Fix For: Slider 0.60 in SLIDER-622,(commits d5569711/ b2768 ) the code to delete a ZK node on cluster destroy was lost. restate -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-650) regression: local zk nodes not being deleted on instance destroy
[ https://issues.apache.org/jira/browse/SLIDER-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SLIDER-650. --- Resolution: Fixed regression: local zk nodes not being deleted on instance destroy Key: SLIDER-650 URL: https://issues.apache.org/jira/browse/SLIDER-650 Project: Slider Issue Type: Bug Components: client Affects Versions: Slider 0.60 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Blocker Fix For: Slider 0.60 in SLIDER-622,(commits d5569711/ b2768 ) the code to delete a ZK node on cluster destroy was lost. restate -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-651) intermittent failure of TestStandaloneAgentAM on windows
Steve Loughran created SLIDER-651:
-
Summary: intermittent failure of TestStandaloneAgentAM on windows
Key: SLIDER-651
URL: https://issues.apache.org/jira/browse/SLIDER-651
Project: Slider
Issue Type: Bug
Components: test
Affects Versions: Slider 0.60
Environment: windows
Reporter: Steve Loughran
Priority: Minor
{{TestStandaloneAgentAM}} failing intermittently saying the port allocated is
out of the range of requested ports.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[VOTE] Apache Slider Incubating Release 0.60.0-incubating RC1
Hi The updated RC1 release of slider is up for review and vote. Please download and review it all changes in this release are listed at: https://issues.apache.org/jira/browse/SLIDER/fixforversion/12327198/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel Main changes since the RC0 - fixed license problem - SLIDER-647: allocation requests not being satisfied when a cluster goes to labels (the default placement policy is now lax, you can request strict on an application instance or component-by-component basis - - SLIDER-650 regression: local zk nodes not being deleted on instance destroy (well spotted Sumit!) Note that this is a source only release and we are voting on the source. artifacts at http://people.apache.org/~stevel/slider/slider-release-0.60.0-incubating-rc1 source at https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;a=shortlog;h=refs/tags/release-0.60.0-incubating-rc0 PGP keys at http://pgp.mit.edu:11371/pks/lookup?op=vindexsearch=ste...@apache.org Build instructions at: http://slider.incubator.apache.org/developing/building.html Vote will be open for 72 hours [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and reason why) -Steve -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
git commit for the slider-0.60.0 RC1
There's some discussion in general@incubator that a git commit must be supplied for a release, not a branch or (movable) tag. the commit used for this release is: c2c06d3 -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (SLIDER-544) Release Slider 0.6.0
[ https://issues.apache.org/jira/browse/SLIDER-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212714#comment-14212714 ] ASF subversion and git services commented on SLIDER-544: Commit 1639762 from [~ste...@apache.org] in branch 'site/trunk' [ https://svn.apache.org/r1639762 ] SLIDER-544 notes in sync with SLIDER-0.60.0-RC1 Release Slider 0.6.0 Key: SLIDER-544 URL: https://issues.apache.org/jira/browse/SLIDER-544 Project: Slider Issue Type: Task Components: build Reporter: Steve Loughran Assignee: Steve Loughran Fix For: Slider 0.70 Original Estimate: 4h Remaining Estimate: 4h -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Do I have to repackage if I change appConfig.json/resources.json
Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks!
Re: Do I have to repackage if I change appConfig.json/resources.json
No, those files are only needed as inputs to create command. The ZIP package is expected to have default versions of the file as a reference - it does not need them for operation. In the current develop or releases/slider-60 we have modified the names to have the word default. On Fri, Nov 14, 2014 at 11:54 AM, hsy...@gmail.com hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Do I have to repackage if I change appConfig.json/resources.json
Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Do I have to repackage if I change appConfig.json/resources.json
So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Do I have to repackage if I change appConfig.json/resources.json
Nope. This is what I do typically. * Get an application package zip file * Extract the appConfig-default.json and resources-default.json from the package * Rename the files to appConfig.json and resources.json respectively and edit as needed * Use install-package to upload the application package zip file * Call create with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Apache Slider Incubating Release 0.60.0-incubating RC1
+1. Modify the following to rc1, source at https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;a=shortlog;h=refs/tags/release-0.60.0-incubating-rc0 Verified SHA and MD5 for both the tar ball and the zip file. The pgp signatures also verify against the public key. Created a clean VM and used the tar ball to successfully build and run unit tests. The tar ball builds fine and all unit tests pass. I used mvn clean -X -Dslider.conf.dir=/usr/work/tar/apache-slider-0.60.0-incubating/src/test/clusters/offline/slider/ package -Prat which is equivalent of what jenkins uses. A note may be added for the Incubator General vote email that the file ./apache-slider-0.60.0-incubating/slider-core/src/test/python/appdef_1.tar is not a binary file. Earlier it was raised as a question in case it is a binary file. thanks -Sumit On Fri, Nov 14, 2014 at 11:26 AM, Steve Loughran ste...@hortonworks.com wrote: Hi The updated RC1 release of slider is up for review and vote. Please download and review it all changes in this release are listed at: https://issues.apache.org/jira/browse/SLIDER/fixforversion/12327198/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel Main changes since the RC0 - fixed license problem - SLIDER-647: allocation requests not being satisfied when a cluster goes to labels (the default placement policy is now lax, you can request strict on an application instance or component-by-component basis - - SLIDER-650 regression: local zk nodes not being deleted on instance destroy (well spotted Sumit!) Note that this is a source only release and we are voting on the source. artifacts at http://people.apache.org/~stevel/slider/slider-release-0.60.0-incubating-rc1 source at https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;a=shortlog;h=refs/tags/release-0.60.0-incubating-rc0 PGP keys at http://pgp.mit.edu:11371/pks/lookup?op=vindexsearch=ste...@apache.org Build instructions at: http://slider.incubator.apache.org/developing/building.html Vote will be open for 72 hours [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and reason why) -Steve -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Do I have to repackage if I change appConfig.json/resources.json
I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/ add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) Thank you! On Fri, Nov 14, 2014 at 2:14 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Nope. This is what I do typically. * Get an application package zip file * Extract the appConfig-default.json and resources-default.json from the package * Rename the files to appConfig.json and resources.json respectively and edit as needed * Use install-package to upload the application package zip file * Call create with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
How to change log level for Slider AM?
How to change log level for Slider AM? Thanks!
Re: Do I have to repackage if I change appConfig.json/resources.json
I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? The file in the .zip package is not used by the running application. The files that are used are the ones that are provided by command line when create is called. *We suggest adding sample defaults to the .zip file as reference so that there are only one file to download for the users.* Which branch are you using? (for me to answer the next questions) On Fri, Nov 14, 2014 at 3:47 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/ add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) Thank you! On Fri, Nov 14, 2014 at 2:14 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Nope. This is what I do typically. * Get an application package zip file * Extract the appConfig-default.json and resources-default.json from the package * Rename the files to appConfig.json and resources.json respectively and edit as needed * Use install-package to upload the application package zip file * Call create with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Do I have to repackage if I change appConfig.json/resources.json
I'm actually using 0.40 On Fri, Nov 14, 2014 at 4:21 PM, Sumit Mohanty smoha...@hortonworks.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? The file in the .zip package is not used by the running application. The files that are used are the ones that are provided by command line when create is called. *We suggest adding sample defaults to the .zip file as reference so that there are only one file to download for the users.* Which branch are you using? (for me to answer the next questions) On Fri, Nov 14, 2014 at 3:47 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/ add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) Thank you! On Fri, Nov 14, 2014 at 2:14 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Nope. This is what I do typically. * Get an application package zip file * Extract the appConfig-default.json and resources-default.json from the package * Rename the files to appConfig.json and resources.json respectively and edit as needed * Use install-package to upload the application package zip file * Call create with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Question about behavior after container failures
Hi all, In the document, I got that slider will try to recover on container failure. But in my test application it doesn't I'm using the 0.40 release. build from source Here is what I get. *If I kill the child process * The agent check_process_status method will raise ComponentIsNotRunning and never get back again 2014-11-14 16:18:40,274 - Error while executing command 'status': Traceback (most recent call last): File /yarn/nm/usercache/vagrant/appcache/application_1415305968048_0008/container_1415305968048_0008_01_02/infra/agent/slider-agent/resource_management/libraries/script/script.py, line 114, in execute method(env) File /yarn/nm/usercache/vagrant/appcache/application_1415305968048_0008/container_1415305968048_0008_01_02/app/definition/package/scripts/kafka.py, line 60, in status check_process_status(status_params.pid_file) File /yarn/nm/usercache/vagrant/appcache/application_1415305968048_0008/container_1415305968048_0008_01_02/infra/agent/slider-agent/resource_management/libraries/functions/check_process_status.py, line 45, in check_process_status raise ComponentIsNotRunning() ComponentIsNotRunning *If I kill the agent process*, it's just silently gone Best, Siyuan
Re: Do I have to repackage if I change appConfig.json/resources.json
Is it possible for you to pick up the latest RC (0.60)? I can help you update the existing application package, if needed. Meanwhile, for 0.40, the log4j.properties file in slider-client-install-dir/conf can be modified to up the log level and the create call will push the file as a localized file for the AppMaster. Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) increase the memory This is not possible when the app is running. Yarn does have a support for that. You need to stop the application and then modify the values. After topping the application you need to edit the resources.json file in HDFS to modify the value (e.g. ./user/USER1/slider/cluster/CLUSTER1/resources.json). add more instances for example Flex command can be used while the application is running. want the application recover from last snapshot If you stop and then start (probably it is freeze and thaw in 0.40) the application then it will use the hostnames that it remembered. -Sumit On Fri, Nov 14, 2014 at 4:35 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm actually using 0.40 On Fri, Nov 14, 2014 at 4:21 PM, Sumit Mohanty smoha...@hortonworks.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? The file in the .zip package is not used by the running application. The files that are used are the ones that are provided by command line when create is called. *We suggest adding sample defaults to the .zip file as reference so that there are only one file to download for the users.* Which branch are you using? (for me to answer the next questions) On Fri, Nov 14, 2014 at 3:47 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/ add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) Thank you! On Fri, Nov 14, 2014 at 2:14 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Nope. This is what I do typically. * Get an application package zip file * Extract the appConfig-default.json and resources-default.json from the package * Rename the files to appConfig.json and resources.json respectively and edit as needed * Use install-package to upload the application package zip file * Call create with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is
Re: Do I have to repackage if I change appConfig.json/resources.json
The only problem is I have to upgrade to hadoop 2.6? On Fri, Nov 14, 2014 at 5:08 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Is it possible for you to pick up the latest RC (0.60)? I can help you update the existing application package, if needed. Meanwhile, for 0.40, the log4j.properties file in slider-client-install-dir/conf can be modified to up the log level and the create call will push the file as a localized file for the AppMaster. Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) increase the memory This is not possible when the app is running. Yarn does have a support for that. You need to stop the application and then modify the values. After topping the application you need to edit the resources.json file in HDFS to modify the value (e.g. ./user/USER1/slider/cluster/CLUSTER1/resources.json). add more instances for example Flex command can be used while the application is running. want the application recover from last snapshot If you stop and then start (probably it is freeze and thaw in 0.40) the application then it will use the hostnames that it remembered. -Sumit On Fri, Nov 14, 2014 at 4:35 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm actually using 0.40 On Fri, Nov 14, 2014 at 4:21 PM, Sumit Mohanty smoha...@hortonworks.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? The file in the .zip package is not used by the running application. The files that are used are the ones that are provided by command line when create is called. *We suggest adding sample defaults to the .zip file as reference so that there are only one file to download for the users.* Which branch are you using? (for me to answer the next questions) On Fri, Nov 14, 2014 at 3:47 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/ add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) Thank you! On Fri, Nov 14, 2014 at 2:14 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Nope. This is what I do typically. * Get an application package zip file * Extract the appConfig-default.json and resources-default.json from the package * Rename the files to appConfig.json and resources.json respectively and edit as needed * Use install-package to upload the application package zip file * Call create with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error,
Re: Do I have to repackage if I change appConfig.json/resources.json
This is not possible when the app is running. Yarn does have a support for that. You need to stop the application and then modify the values. After topping the application you need to edit the resources.json file in HDFS to modify the value (e.g. ./user/USER1/slider/cluster/CLUSTER1/resources.json). This is fine. I don't need to change this at run time, but I don't want to recreate either. Can I just stop, edit and start the application again with new properties been picked up? If you stop and then start (probably it is freeze and thaw in 0.40) the application then it will use the hostnames that it remembered. What if I use *yarn application -kill* to stop the application? On Fri, Nov 14, 2014 at 5:09 PM, hsy...@gmail.com hsy...@gmail.com wrote: The only problem is I have to upgrade to hadoop 2.6? On Fri, Nov 14, 2014 at 5:08 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Is it possible for you to pick up the latest RC (0.60)? I can help you update the existing application package, if needed. Meanwhile, for 0.40, the log4j.properties file in slider-client-install-dir/conf can be modified to up the log level and the create call will push the file as a localized file for the AppMaster. Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) increase the memory This is not possible when the app is running. Yarn does have a support for that. You need to stop the application and then modify the values. After topping the application you need to edit the resources.json file in HDFS to modify the value (e.g. ./user/USER1/slider/cluster/CLUSTER1/resources.json). add more instances for example Flex command can be used while the application is running. want the application recover from last snapshot If you stop and then start (probably it is freeze and thaw in 0.40) the application then it will use the hostnames that it remembered. -Sumit On Fri, Nov 14, 2014 at 4:35 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm actually using 0.40 On Fri, Nov 14, 2014 at 4:21 PM, Sumit Mohanty smoha...@hortonworks.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? The file in the .zip package is not used by the running application. The files that are used are the ones that are provided by command line when create is called. *We suggest adding sample defaults to the .zip file as reference so that there are only one file to download for the users.* Which branch are you using? (for me to answer the next questions) On Fri, Nov 14, 2014 at 3:47 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/ add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) Thank you! On Fri, Nov 14, 2014 at 2:14 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Nope. This is what I do typically. * Get an application package zip file * Extract the appConfig-default.json and resources-default.json from the package * Rename the files to appConfig.json and resources.json respectively and edit as needed * Use install-package to upload the application package zip file * Call create with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources resources file path the app config references the application package in HDFS, which can be pre-seeded using “slider install-package” — Jon On Nov 14, 2014, at 2:54 PM, hsy...@gmail.com wrote:
Re: Question about behavior after container failures
*If I kill the child process * So are you observing that a replacement container instance is never started? Can you check the AM log? What is expected to happen here is as follows: - process goes down, and then agent goes down, AM notices, asks for a new container from Yarn. if you want the agent to restart the process then you need to set autoStartOnFailuretrue/autoStartOnFailure for the component in the metainfo.xml. This is not part of 0.40 though. *If I kill the agent process* This is YARN-1922 (https://issues.apache.org/jira/browse/YARN-1922). On Fri, Nov 14, 2014 at 4:46 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi all, In the document, I got that slider will try to recover on container failure. But in my test application it doesn't I'm using the 0.40 release. build from source Here is what I get. *If I kill the child process * The agent check_process_status method will raise ComponentIsNotRunning and never get back again 2014-11-14 16:18:40,274 - Error while executing command 'status': Traceback (most recent call last): File /yarn/nm/usercache/vagrant/appcache/application_1415305968048_0008/container_1415305968048_0008_01_02/infra/agent/slider-agent/resource_management/libraries/script/script.py, line 114, in execute method(env) File /yarn/nm/usercache/vagrant/appcache/application_1415305968048_0008/container_1415305968048_0008_01_02/app/definition/package/scripts/kafka.py, line 60, in status check_process_status(status_params.pid_file) File /yarn/nm/usercache/vagrant/appcache/application_1415305968048_0008/container_1415305968048_0008_01_02/infra/agent/slider-agent/resource_management/libraries/functions/check_process_status.py, line 45, in check_process_status raise ComponentIsNotRunning() ComponentIsNotRunning *If I kill the agent process*, it's just silently gone Best, Siyuan -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Do I have to repackage if I change appConfig.json/resources.json
On Fri, Nov 14, 2014 at 5:14 PM, hsy...@gmail.com hsy...@gmail.com wrote: This is not possible when the app is running. Yarn does have a support for that. You need to stop the application and then modify the values. After topping the application you need to edit the resources.json file in HDFS to modify the value (e.g. ./user/USER1/slider/cluster/CLUSTER1/resources.json). This is fine. I don't need to change this at run time, but I don't want to recreate either. Can I just stop, edit and start the application again with new properties been picked up? Yes. If you stop and then start (probably it is freeze and thaw in 0.40) the application then it will use the hostnames that it remembered. What if I use *yarn application -kill* to stop the application? Should be OK. On Fri, Nov 14, 2014 at 5:09 PM, hsy...@gmail.com hsy...@gmail.com wrote: The only problem is I have to upgrade to hadoop 2.6? On Fri, Nov 14, 2014 at 5:08 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Is it possible for you to pick up the latest RC (0.60)? I can help you update the existing application package, if needed. Meanwhile, for 0.40, the log4j.properties file in slider-client-install-dir/conf can be modified to up the log level and the create call will push the file as a localized file for the AppMaster. Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) increase the memory This is not possible when the app is running. Yarn does have a support for that. You need to stop the application and then modify the values. After topping the application you need to edit the resources.json file in HDFS to modify the value (e.g. ./user/USER1/slider/cluster/CLUSTER1/resources.json). add more instances for example Flex command can be used while the application is running. want the application recover from last snapshot If you stop and then start (probably it is freeze and thaw in 0.40) the application then it will use the hostnames that it remembered. -Sumit On Fri, Nov 14, 2014 at 4:35 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm actually using 0.40 On Fri, Nov 14, 2014 at 4:21 PM, Sumit Mohanty smoha...@hortonworks.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? The file in the .zip package is not used by the running application. The files that are used are the ones that are provided by command line when create is called. *We suggest adding sample defaults to the .zip file as reference so that there are only one file to download for the users.* Which branch are you using? (for me to answer the next questions) On Fri, Nov 14, 2014 at 3:47 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig-default.json and resource-default.json is outdated? Another question is actually I'm asking how to change the configuration without recreate the cluster. Sometime I just want to increase the memory/ add more instances for example and I want the application recover from last snapshot(run on same machines it allocated before) Thank you! On Fri, Nov 14, 2014 at 2:14 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Nope. This is what I do typically. * Get an application package zip file * Extract the appConfig-default.json and resources-default.json from the package * Rename the files to appConfig.json and resources.json respectively and edit as needed * Use install-package to upload the application package zip file * Call create with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma...@hortonworks.com wrote: Perhaps I’m misunderstanding your question, but in general, if making modifications to those files, you can simply create a new application instance referencing the new versions of the file from the command line: ./slider create app name —template appConfig path —resources
Re: How to change log level for Slider AM?
Under conf directory there is a file named *log4j-server.properties*. Just edit that file and change *log4j.rootLogger* from INFO to what you like. Then do slider stop and slider start. The new level for Slider AM will take effect. The *slider.log* file is rotated as well on a 100KB size threshold. This change was made fairly recently so you need to be on the latest develop or releases/slider-0.60 branch. -Gour On Fri, Nov 14, 2014 at 3:57 PM, hsy...@gmail.com hsy...@gmail.com wrote: How to change log level for Slider AM? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
