Version 0.9.0 does not have the gc executor. Version 0.9.0 uses Mesos' task reconciliation <http://mesos.apache.org/documentation/latest/reconciliation/> API instead.
On Wed, Oct 21, 2015 at 9:28 AM, Krish <[email protected]> wrote: > Thanks Bill for the location to the debs. I was finally able to run > aurora. :) > > I did find thermos_executor.pex & thermos_observer after installing > aurora-executor. I still could not find gc_executor.pex on my system. > Is there a location from where I can download the binaries for *.pex or > build them from scratch? > > root@dev:/# find . -name "*.pex" > ./usr/share/aurora/bin/thermos_executor.pex > ./usr/share/aurora/bin/kaurora_admin.pex > ./usr/share/aurora/bin/kaurora.pex > ./usr/share/aurora/bin/thermos.pex > ./usr/share/aurora/bin/thermos_observer.pex > ./home/ubuntu/.pex > ./root/.pex > > > > > > -- > κρισhναν > > On Tue, Oct 20, 2015 at 11:46 PM, Bill Farner <[email protected]> wrote: > >> Aurora currently requires an executor, so setting it to /dev/null will >> not work. Happy to talk further about your thoughts around sidestepping >> the executor. >> >> As for working with the scheduler source code, it's a standard gradle >> project and we tend to use intellij. Docs to help ramp on that: >> https://github.com/apache/aurora/blob/master/docs/developing-aurora-scheduler.md >> >> As for builds - the .zip is a source distribution, so it won't have any >> pre-built binaries. If you're on debian, we have official debs here: >> https://bintray.com/apache/aurora >> You can see how they're built here (and can build your own) packages: >> https://github.com/apache/aurora-packaging >> We're close to having official RPMs, but none to speak of yet. >> >> On Tue, Oct 20, 2015 at 9:47 AM, Krish <[email protected]> wrote: >> >>> Stephen, >>> I am trying to get started and run aurora without thermos executor >>> (setting it to /dev/null does not help) - on a local linux box for now & >>> planning to containerize/dockerize it later. >>> >>> Can you please point me to the right documentation (or a pointer to the >>> cli parsing source code) which can help me resolve this? Also, are there >>> any steps steps to import source code into eclipse to browse & analyze code >>> for this. >>> >>> Also, where do i find all the *.pex files? They are not present in the >>> zip file nor anywhere in the built source code. >>> >>> I know I am asking too many queries on a single thread here, & would >>> appreciate the help. >>> I think at the end of this, I will put the steps I followed in a >>> gist/blog so others might find their way around, & not struggle as much. >>> >>> >>> >>> -- >>> κρισhναν >>> >>> On Tue, Oct 20, 2015 at 4:09 PM, Erb, Stephan < >>> [email protected]> wrote: >>> >>>> Hi Krish, >>>> >>>> >>>> you don't have to set framework_authentication_file and >>>> zk_digest_credentials. The scheduler help text is misleading here as >>>> everything will work fine if you leave those empty. >>>> >>>> >>>> In addition, looks like you are misunderstanding the usage of the >>>> thermos_executor_path command line flag of the scheduler. It is >>>> supposed to point to the binary containing the generic Aurora executor >>>> (thermos_executor.pex). You only need the hello_world.aurora once >>>> your scheduler is up an running. It serves as an example input for the >>>> aurora command line client which can be used to scheduler jobs and services >>>> on an Aurora master. >>>> >>>> >>>> Have you tried to use the vagrant box? Just type 'vagrant up`in a >>>> checkout of the Aurora source code. It gives you a running scheduler to >>>> play with. Once you have understood how it works, you can start trying to >>>> install it on your own (by reverse-engineering the vagrant box). >>>> >>>> >>>> Hope this helps a little, >>>> >>>> Stephan >>>> >>>> >>>> >>>> ------------------------------ >>>> *From:* Krish <[email protected]> >>>> *Sent:* Tuesday, October 20, 2015 11:39 AM >>>> *To:* Bill Farner >>>> *Cc:* [email protected]; Erb, Stephan >>>> >>>> *Subject:* Re: Stacktrace when running Apache Aurora >>>> >>>> Bill/Stephen, >>>> I still get a stacktrace when running the aurora scheduler CLI. >>>> >>>> I do not know what to specify for -framework_authentication_file >>>> & -zk_digest_credentials, and they are required arguments. >>>> >>>> I am not using any authentication on Mesos master, do I still need the >>>> framework_authentication_file parameter? >>>> >>>> >>>> rm -rf /db /backup_dir >>>> mesos-log initialize --path="/db" >>>> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/ >>>> JAVA_OPTS="-Xmx1536m -Xms256m" >>>> /usr/local/aurora-scheduler/bin/aurora-scheduler -backup_dir=/backup_dir >>>> -cluster_name=tc -mesos_master_address=zk://localhost:2181/mesos/master >>>> -serverset_path=/scheduler/aurora -zk_endpoints=localhost:2181 >>>> -native_log_quorum_size=1 -vlog=SEVERE -logtostderr=false >>>> -native_log_file_path=/db >>>> -thermos_executor_path=/home/ubuntu/hello_world.aurora >>>> ... >>>> ... >>>> INFO: Binding org.apache.aurora.scheduler.http.Utilization to >>>> GuiceManagedCompon >>>> entProvider with the scope "PerRequest" >>>> Oct 20, 2015 9:27:40 AM >>>> org.apache.aurora.scheduler.cron.quartz.CronModule provi >>>> deTimeZone >>>> WARNING: Cron schedules are configured to fire according to timezone >>>> Greenwich M >>>> ean Time but system timezone is set to Coordinated Universal Time >>>> Oct 20, 2015 9:27:41 AM org.eclipse.jetty.server.AbstractConnector >>>> doStart >>>> INFO: Started [email protected]:43843 >>>> E1020 09:27:41.290 THREAD1 >>>> org.apache.aurora.scheduler.SchedulerLifecycle$9.exec >>>> ute: Caught unchecked exception: com.google.inject.ProvisionException: >>>> Guice pro >>>> vision errors: >>>> >>>> 1) Error in custom provider, java.lang.IllegalArgumentException: Path >>>> cannot be null at >>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >>>> LogStreamModule.java:117) >>>> at >>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >>>> LogStreamModule.java:117) >>>> while locating org.apache.mesos.Log >>>> at >>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLogInterf >>>> ace(MesosLogStreamModule.java:152) >>>> while locating org.apache.aurora.scheduler.log.mesos.LogInterface >>>> >>>> 1 error >>>> com.google.inject.ProvisionException: Guice provision errors: >>>> >>>> 1) Error in custom provider, java.lang.IllegalArgumentException: Path >>>> cannot be >>>> null >>>> at >>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >>>> LogStreamModule.java:117) >>>> at >>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >>>> LogStreamModule.java:117) >>>> while locating org.apache.mesos.Log >>>> at >>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLogInterf >>>> ace(MesosLogStreamModule.java:152) >>>> while locating org.apache.aurora.scheduler.log.mesos.LogInterface >>>> >>>> 1 error >>>> at >>>> com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987) >>>> at >>>> org.apache.aurora.scheduler.log.mesos.MesosLog.open(MesosLog.java:136 >>>> >>>> >>>> >>>> -- >>>> κρισhναν >>>> >>>> On Tue, Oct 20, 2015 at 6:14 AM, Bill Farner <[email protected]> >>>> wrote: >>>> >>>>> The typical flow is that you keep your .aurora file checked into git, >>>>> and commit every time you deploy/update. When you change your file, you >>>>> will instruct Aurora to update the live job (have a look at aurora >>>>> update -h). Aurora will perform a rolling upgrade of your job to the >>>>> new config. You'll use this same flow for updating your job's software as >>>>> well as resizing the job. >>>>> >>>>> For (3), you could set up alerting for stats that the scheduler >>>>> exports. Have a look here for monitoring background: >>>>> https://github.com/apache/aurora/blob/master/docs/monitoring.md >>>>> >>>>> You'll find want to look at scheduler stats related to 'pending'. >>>>> >>>>> On Mon, Oct 19, 2015 at 12:16 PM, Krish <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks for the pointer. Now I notice that the aurora-scheduler script >>>>>> has the --thermos_executor_path as a mandatory requirement. >>>>>> >>>>>> I have a couple of questions on how the thermos_executor/.aurora >>>>>> config file functions: >>>>>> 1. Do we have to statically define the file beforehand? >>>>>> 2. What happens when we want to dynamically change the config, say >>>>>> increasing the number of instances of a service required? Does aurora >>>>>> require a reboot then? >>>>>> 3. How do I get notified about the message mesos sends when it cannot >>>>>> schedule tasks for lack of resources? Should I depend on aurora for this >>>>>> or >>>>>> try to look for a hook into mesos? >>>>>> >>>>>> I think a little bit of context would help here. >>>>>> What I plan to check is to run a very basic job/task inside a docker >>>>>> container with aurora & wait for a 'resource not available' message from >>>>>> mesos, and accordingly call an api to spin up a new node in my cluster. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> κρισhναν >>>>>> >>>>>> On Mon, Oct 19, 2015 at 1:24 PM, Erb, Stephan < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I believe you are missing the thermos_executor options that have to >>>>>>> be passed to the scheduler command line. >>>>>>> >>>>>>> >>>>>>> See >>>>>>> https://github.com/apache/aurora/blob/33d7e2170a86f54722a02a2dc9cb1e09fb52df25/examples/vagrant/upstart/aurora-scheduler.conf#L39 >>>>>>> for an example >>>>>>> >>>>>>> >>>>>>> Best Regards, >>>>>>> >>>>>>> Stephan >>>>>>> >>>>>>> >>>>>>> ------------------------------ >>>>>>> *From:* Krish <[email protected]> >>>>>>> *Sent:* Monday, October 19, 2015 8:45 AM >>>>>>> *To:* [email protected] >>>>>>> *Subject:* Re: Stacktrace when running Apache Aurora >>>>>>> >>>>>>> Hi, >>>>>>> I am a n00b with apache aurora & trying to experiment some things on >>>>>>> my local machine with zookeeper and mesos-master running locally. They >>>>>>> have >>>>>>> initialized properly. When I try to run aurora with the required >>>>>>> options, I >>>>>>> get the following error, & googing hasn't helped me much here. >>>>>>> Appreciate any help. Thanks in advance. >>>>>>> >>>>>>> ... >>>>>>> ... >>>>>>> WARNING: Method [public void >>>>>>> org.apache.aurora.scheduler.storage.log.SnapshotStoreImpl.applySnapshot(java.lang.Object)] >>>>>>> is synthetic and is being intercepted by >>>>>>> [com.twitter.common.inject.TimedInterceptor@604c5de8]. This could >>>>>>> indicate a bug. The method >>>>>>> may be intercepted twice, or may not be intercepted at all. >>>>>>> Exception in thread "main" com.google.inject.CreationException: >>>>>>> Guice creation errors: >>>>>>> >>>>>>> 1) An exception was caught and reported. Message: A value may only >>>>>>> be retrieved from a variable that has a default or has been >>>>>>> set. >>>>>>> at >>>>>>> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133) >>>>>>> >>>>>>> 2) Could not find a suitable constructor in >>>>>>> org.apache.aurora.scheduler.mesos.ExecutorSettings. Classes must have >>>>>>> either one (a >>>>>>> nd only one) constructor annotated with @Inject or a zero-argument >>>>>>> constructor that is not private. >>>>>>> at >>>>>>> org.apache.aurora.scheduler.mesos.ExecutorSettings.class(ExecutorSettings.java:43) >>>>>>> at >>>>>>> org.apache.aurora.scheduler.app.SchedulerMain$3.configure(SchedulerMain.java:204) >>>>>>> >>>>>>> 2 errors >>>>>>> at >>>>>>> com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) >>>>>>> at >>>>>>> com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) >>>>>>> at >>>>>>> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) >>>>>>> at com.google.inject.Guice.createInjector(Guice.java:95) >>>>>>> at com.google.inject.Guice.createInjector(Guice.java:83) >>>>>>> at >>>>>>> com.twitter.common.application.AppLauncher.configureInjection(AppLauncher.java:120) >>>>>>> at >>>>>>> com.twitter.common.application.AppLauncher.run(AppLauncher.java:87) >>>>>>> at >>>>>>> com.twitter.common.application.AppLauncher.launch(AppLauncher.java:181) >>>>>>> at >>>>>>> com.twitter.common.application.AppLauncher.launch(AppLauncher.java:142) >>>>>>> at >>>>>>> org.apache.aurora.scheduler.app.SchedulerMain.main(SchedulerMain.java:263) >>>>>>> Caused by: java.lang.IllegalStateException: A value may only be >>>>>>> retrieved from a variable that has a default or has been set. >>>>>>> at >>>>>>> com.google.common.base.Preconditions.checkState(Preconditions.java:176) >>>>>>> at com.twitter.common.args.Arg.get(Arg.java:82) >>>>>>> at >>>>>>> org.apache.aurora.scheduler.app.SchedulerMain$3.configure(SchedulerMain.java:206) >>>>>>> at >>>>>>> com.google.inject.AbstractModule.configure(AbstractModule.java:59) >>>>>>> at >>>>>>> com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223) >>>>>>> at >>>>>>> com.google.inject.util.Modules$2.configure(Modules.java:114) >>>>>>> at >>>>>>> com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223) >>>>>>> at >>>>>>> com.google.inject.spi.Elements.getElements(Elements.java:101) >>>>>>> at >>>>>>> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133) >>>>>>> at >>>>>>> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103) >>>>>>> ... 7 more >>>>>>> >>>>>>> Complete logs are present @http://pastebin.com/i72HvbYi. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> -- >>>>>>>> κρισhναν >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -- Zameer Manji
