Thanks Bill for the location to the debs. I was finally able to run aurora. :)
I did find thermos_executor.pex & thermos_observer after installing aurora-executor. I still could not find gc_executor.pex on my system. Is there a location from where I can download the binaries for *.pex or build them from scratch? root@dev:/# find . -name "*.pex" ./usr/share/aurora/bin/thermos_executor.pex ./usr/share/aurora/bin/kaurora_admin.pex ./usr/share/aurora/bin/kaurora.pex ./usr/share/aurora/bin/thermos.pex ./usr/share/aurora/bin/thermos_observer.pex ./home/ubuntu/.pex ./root/.pex -- κρισhναν On Tue, Oct 20, 2015 at 11:46 PM, Bill Farner <[email protected]> wrote: > Aurora currently requires an executor, so setting it to /dev/null will not > work. Happy to talk further about your thoughts around sidestepping the > executor. > > As for working with the scheduler source code, it's a standard gradle > project and we tend to use intellij. Docs to help ramp on that: > https://github.com/apache/aurora/blob/master/docs/developing-aurora-scheduler.md > > As for builds - the .zip is a source distribution, so it won't have any > pre-built binaries. If you're on debian, we have official debs here: > https://bintray.com/apache/aurora > You can see how they're built here (and can build your own) packages: > https://github.com/apache/aurora-packaging > We're close to having official RPMs, but none to speak of yet. > > On Tue, Oct 20, 2015 at 9:47 AM, Krish <[email protected]> wrote: > >> Stephen, >> I am trying to get started and run aurora without thermos executor >> (setting it to /dev/null does not help) - on a local linux box for now & >> planning to containerize/dockerize it later. >> >> Can you please point me to the right documentation (or a pointer to the >> cli parsing source code) which can help me resolve this? Also, are there >> any steps steps to import source code into eclipse to browse & analyze code >> for this. >> >> Also, where do i find all the *.pex files? They are not present in the >> zip file nor anywhere in the built source code. >> >> I know I am asking too many queries on a single thread here, & would >> appreciate the help. >> I think at the end of this, I will put the steps I followed in a >> gist/blog so others might find their way around, & not struggle as much. >> >> >> >> -- >> κρισhναν >> >> On Tue, Oct 20, 2015 at 4:09 PM, Erb, Stephan < >> [email protected]> wrote: >> >>> Hi Krish, >>> >>> >>> you don't have to set framework_authentication_file and >>> zk_digest_credentials. The scheduler help text is misleading here as >>> everything will work fine if you leave those empty. >>> >>> >>> In addition, looks like you are misunderstanding the usage of the >>> thermos_executor_path command line flag of the scheduler. It is >>> supposed to point to the binary containing the generic Aurora executor >>> (thermos_executor.pex). You only need the hello_world.aurora once your >>> scheduler is up an running. It serves as an example input for the aurora >>> command line client which can be used to scheduler jobs and services on an >>> Aurora master. >>> >>> >>> Have you tried to use the vagrant box? Just type 'vagrant up`in a >>> checkout of the Aurora source code. It gives you a running scheduler to >>> play with. Once you have understood how it works, you can start trying to >>> install it on your own (by reverse-engineering the vagrant box). >>> >>> >>> Hope this helps a little, >>> >>> Stephan >>> >>> >>> >>> ------------------------------ >>> *From:* Krish <[email protected]> >>> *Sent:* Tuesday, October 20, 2015 11:39 AM >>> *To:* Bill Farner >>> *Cc:* [email protected]; Erb, Stephan >>> >>> *Subject:* Re: Stacktrace when running Apache Aurora >>> >>> Bill/Stephen, >>> I still get a stacktrace when running the aurora scheduler CLI. >>> >>> I do not know what to specify for -framework_authentication_file >>> & -zk_digest_credentials, and they are required arguments. >>> >>> I am not using any authentication on Mesos master, do I still need the >>> framework_authentication_file parameter? >>> >>> >>> rm -rf /db /backup_dir >>> mesos-log initialize --path="/db" >>> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/ >>> JAVA_OPTS="-Xmx1536m -Xms256m" >>> /usr/local/aurora-scheduler/bin/aurora-scheduler -backup_dir=/backup_dir >>> -cluster_name=tc -mesos_master_address=zk://localhost:2181/mesos/master >>> -serverset_path=/scheduler/aurora -zk_endpoints=localhost:2181 >>> -native_log_quorum_size=1 -vlog=SEVERE -logtostderr=false >>> -native_log_file_path=/db >>> -thermos_executor_path=/home/ubuntu/hello_world.aurora >>> ... >>> ... >>> INFO: Binding org.apache.aurora.scheduler.http.Utilization to >>> GuiceManagedCompon >>> entProvider with the scope "PerRequest" >>> Oct 20, 2015 9:27:40 AM >>> org.apache.aurora.scheduler.cron.quartz.CronModule provi >>> deTimeZone >>> WARNING: Cron schedules are configured to fire according to timezone >>> Greenwich M >>> ean Time but system timezone is set to Coordinated Universal Time >>> Oct 20, 2015 9:27:41 AM org.eclipse.jetty.server.AbstractConnector >>> doStart >>> INFO: Started [email protected]:43843 >>> E1020 09:27:41.290 THREAD1 >>> org.apache.aurora.scheduler.SchedulerLifecycle$9.exec >>> ute: Caught unchecked exception: com.google.inject.ProvisionException: >>> Guice pro >>> vision errors: >>> >>> 1) Error in custom provider, java.lang.IllegalArgumentException: Path >>> cannot be null at >>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >>> LogStreamModule.java:117) >>> at >>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >>> LogStreamModule.java:117) >>> while locating org.apache.mesos.Log >>> at >>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLogInterf >>> ace(MesosLogStreamModule.java:152) >>> while locating org.apache.aurora.scheduler.log.mesos.LogInterface >>> >>> 1 error >>> com.google.inject.ProvisionException: Guice provision errors: >>> >>> 1) Error in custom provider, java.lang.IllegalArgumentException: Path >>> cannot be >>> null >>> at >>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >>> LogStreamModule.java:117) >>> at >>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >>> LogStreamModule.java:117) >>> while locating org.apache.mesos.Log >>> at >>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLogInterf >>> ace(MesosLogStreamModule.java:152) >>> while locating org.apache.aurora.scheduler.log.mesos.LogInterface >>> >>> 1 error >>> at >>> com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987) >>> at >>> org.apache.aurora.scheduler.log.mesos.MesosLog.open(MesosLog.java:136 >>> >>> >>> >>> -- >>> κρισhναν >>> >>> On Tue, Oct 20, 2015 at 6:14 AM, Bill Farner <[email protected]> wrote: >>> >>>> The typical flow is that you keep your .aurora file checked into git, >>>> and commit every time you deploy/update. When you change your file, you >>>> will instruct Aurora to update the live job (have a look at aurora >>>> update -h). Aurora will perform a rolling upgrade of your job to the >>>> new config. You'll use this same flow for updating your job's software as >>>> well as resizing the job. >>>> >>>> For (3), you could set up alerting for stats that the scheduler >>>> exports. Have a look here for monitoring background: >>>> https://github.com/apache/aurora/blob/master/docs/monitoring.md >>>> >>>> You'll find want to look at scheduler stats related to 'pending'. >>>> >>>> On Mon, Oct 19, 2015 at 12:16 PM, Krish <[email protected]> >>>> wrote: >>>> >>>>> Thanks for the pointer. Now I notice that the aurora-scheduler script >>>>> has the --thermos_executor_path as a mandatory requirement. >>>>> >>>>> I have a couple of questions on how the thermos_executor/.aurora >>>>> config file functions: >>>>> 1. Do we have to statically define the file beforehand? >>>>> 2. What happens when we want to dynamically change the config, say >>>>> increasing the number of instances of a service required? Does aurora >>>>> require a reboot then? >>>>> 3. How do I get notified about the message mesos sends when it cannot >>>>> schedule tasks for lack of resources? Should I depend on aurora for this >>>>> or >>>>> try to look for a hook into mesos? >>>>> >>>>> I think a little bit of context would help here. >>>>> What I plan to check is to run a very basic job/task inside a docker >>>>> container with aurora & wait for a 'resource not available' message from >>>>> mesos, and accordingly call an api to spin up a new node in my cluster. >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> κρισhναν >>>>> >>>>> On Mon, Oct 19, 2015 at 1:24 PM, Erb, Stephan < >>>>> [email protected]> wrote: >>>>> >>>>>> I believe you are missing the thermos_executor options that have to >>>>>> be passed to the scheduler command line. >>>>>> >>>>>> >>>>>> See >>>>>> https://github.com/apache/aurora/blob/33d7e2170a86f54722a02a2dc9cb1e09fb52df25/examples/vagrant/upstart/aurora-scheduler.conf#L39 >>>>>> for an example >>>>>> >>>>>> >>>>>> Best Regards, >>>>>> >>>>>> Stephan >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> *From:* Krish <[email protected]> >>>>>> *Sent:* Monday, October 19, 2015 8:45 AM >>>>>> *To:* [email protected] >>>>>> *Subject:* Re: Stacktrace when running Apache Aurora >>>>>> >>>>>> Hi, >>>>>> I am a n00b with apache aurora & trying to experiment some things on >>>>>> my local machine with zookeeper and mesos-master running locally. They >>>>>> have >>>>>> initialized properly. When I try to run aurora with the required >>>>>> options, I >>>>>> get the following error, & googing hasn't helped me much here. >>>>>> Appreciate any help. Thanks in advance. >>>>>> >>>>>> ... >>>>>> ... >>>>>> WARNING: Method [public void >>>>>> org.apache.aurora.scheduler.storage.log.SnapshotStoreImpl.applySnapshot(java.lang.Object)] >>>>>> is synthetic and is being intercepted by >>>>>> [com.twitter.common.inject.TimedInterceptor@604c5de8]. This could >>>>>> indicate a bug. The method >>>>>> may be intercepted twice, or may not be intercepted at all. >>>>>> Exception in thread "main" com.google.inject.CreationException: Guice >>>>>> creation errors: >>>>>> >>>>>> 1) An exception was caught and reported. Message: A value may only be >>>>>> retrieved from a variable that has a default or has been >>>>>> set. >>>>>> at >>>>>> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133) >>>>>> >>>>>> 2) Could not find a suitable constructor in >>>>>> org.apache.aurora.scheduler.mesos.ExecutorSettings. Classes must have >>>>>> either one (a >>>>>> nd only one) constructor annotated with @Inject or a zero-argument >>>>>> constructor that is not private. >>>>>> at >>>>>> org.apache.aurora.scheduler.mesos.ExecutorSettings.class(ExecutorSettings.java:43) >>>>>> at >>>>>> org.apache.aurora.scheduler.app.SchedulerMain$3.configure(SchedulerMain.java:204) >>>>>> >>>>>> 2 errors >>>>>> at >>>>>> com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) >>>>>> at >>>>>> com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) >>>>>> at >>>>>> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) >>>>>> at com.google.inject.Guice.createInjector(Guice.java:95) >>>>>> at com.google.inject.Guice.createInjector(Guice.java:83) >>>>>> at >>>>>> com.twitter.common.application.AppLauncher.configureInjection(AppLauncher.java:120) >>>>>> at >>>>>> com.twitter.common.application.AppLauncher.run(AppLauncher.java:87) >>>>>> at >>>>>> com.twitter.common.application.AppLauncher.launch(AppLauncher.java:181) >>>>>> at >>>>>> com.twitter.common.application.AppLauncher.launch(AppLauncher.java:142) >>>>>> at >>>>>> org.apache.aurora.scheduler.app.SchedulerMain.main(SchedulerMain.java:263) >>>>>> Caused by: java.lang.IllegalStateException: A value may only be >>>>>> retrieved from a variable that has a default or has been set. >>>>>> at >>>>>> com.google.common.base.Preconditions.checkState(Preconditions.java:176) >>>>>> at com.twitter.common.args.Arg.get(Arg.java:82) >>>>>> at >>>>>> org.apache.aurora.scheduler.app.SchedulerMain$3.configure(SchedulerMain.java:206) >>>>>> at >>>>>> com.google.inject.AbstractModule.configure(AbstractModule.java:59) >>>>>> at >>>>>> com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223) >>>>>> at >>>>>> com.google.inject.util.Modules$2.configure(Modules.java:114) >>>>>> at >>>>>> com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223) >>>>>> at >>>>>> com.google.inject.spi.Elements.getElements(Elements.java:101) >>>>>> at >>>>>> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133) >>>>>> at >>>>>> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103) >>>>>> ... 7 more >>>>>> >>>>>> Complete logs are present @http://pastebin.com/i72HvbYi. >>>>>> >>>>>> >>>>>> >>>>>>> -- >>>>>>> κρισhναν >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
