Aurora currently requires an executor, so setting it to /dev/null will not work. Happy to talk further about your thoughts around sidestepping the executor.
As for working with the scheduler source code, it's a standard gradle project and we tend to use intellij. Docs to help ramp on that: https://github.com/apache/aurora/blob/master/docs/developing-aurora-scheduler.md As for builds - the .zip is a source distribution, so it won't have any pre-built binaries. If you're on debian, we have official debs here: https://bintray.com/apache/aurora You can see how they're built here (and can build your own) packages: https://github.com/apache/aurora-packaging We're close to having official RPMs, but none to speak of yet. On Tue, Oct 20, 2015 at 9:47 AM, Krish <[email protected]> wrote: > Stephen, > I am trying to get started and run aurora without thermos executor > (setting it to /dev/null does not help) - on a local linux box for now & > planning to containerize/dockerize it later. > > Can you please point me to the right documentation (or a pointer to the > cli parsing source code) which can help me resolve this? Also, are there > any steps steps to import source code into eclipse to browse & analyze code > for this. > > Also, where do i find all the *.pex files? They are not present in the zip > file nor anywhere in the built source code. > > I know I am asking too many queries on a single thread here, & would > appreciate the help. > I think at the end of this, I will put the steps I followed in a gist/blog > so others might find their way around, & not struggle as much. > > > > -- > κρισhναν > > On Tue, Oct 20, 2015 at 4:09 PM, Erb, Stephan <[email protected] > > wrote: > >> Hi Krish, >> >> >> you don't have to set framework_authentication_file and >> zk_digest_credentials. The scheduler help text is misleading here as >> everything will work fine if you leave those empty. >> >> >> In addition, looks like you are misunderstanding the usage of the >> thermos_executor_path command line flag of the scheduler. It is supposed >> to point to the binary containing the generic Aurora executor >> (thermos_executor.pex). You only need the hello_world.aurora once your >> scheduler is up an running. It serves as an example input for the aurora >> command line client which can be used to scheduler jobs and services on an >> Aurora master. >> >> >> Have you tried to use the vagrant box? Just type 'vagrant up`in a >> checkout of the Aurora source code. It gives you a running scheduler to >> play with. Once you have understood how it works, you can start trying to >> install it on your own (by reverse-engineering the vagrant box). >> >> >> Hope this helps a little, >> >> Stephan >> >> >> >> ------------------------------ >> *From:* Krish <[email protected]> >> *Sent:* Tuesday, October 20, 2015 11:39 AM >> *To:* Bill Farner >> *Cc:* [email protected]; Erb, Stephan >> >> *Subject:* Re: Stacktrace when running Apache Aurora >> >> Bill/Stephen, >> I still get a stacktrace when running the aurora scheduler CLI. >> >> I do not know what to specify for -framework_authentication_file >> & -zk_digest_credentials, and they are required arguments. >> >> I am not using any authentication on Mesos master, do I still need the >> framework_authentication_file parameter? >> >> >> rm -rf /db /backup_dir >> mesos-log initialize --path="/db" >> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/ >> JAVA_OPTS="-Xmx1536m -Xms256m" >> /usr/local/aurora-scheduler/bin/aurora-scheduler -backup_dir=/backup_dir >> -cluster_name=tc -mesos_master_address=zk://localhost:2181/mesos/master >> -serverset_path=/scheduler/aurora -zk_endpoints=localhost:2181 >> -native_log_quorum_size=1 -vlog=SEVERE -logtostderr=false >> -native_log_file_path=/db >> -thermos_executor_path=/home/ubuntu/hello_world.aurora >> ... >> ... >> INFO: Binding org.apache.aurora.scheduler.http.Utilization to >> GuiceManagedCompon >> entProvider with the scope "PerRequest" >> Oct 20, 2015 9:27:40 AM >> org.apache.aurora.scheduler.cron.quartz.CronModule provi >> deTimeZone >> WARNING: Cron schedules are configured to fire according to timezone >> Greenwich M >> ean Time but system timezone is set to Coordinated Universal Time >> Oct 20, 2015 9:27:41 AM org.eclipse.jetty.server.AbstractConnector doStart >> INFO: Started [email protected]:43843 >> E1020 09:27:41.290 THREAD1 >> org.apache.aurora.scheduler.SchedulerLifecycle$9.exec >> ute: Caught unchecked exception: com.google.inject.ProvisionException: >> Guice pro >> vision errors: >> >> 1) Error in custom provider, java.lang.IllegalArgumentException: Path >> cannot be null at >> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >> LogStreamModule.java:117) >> at >> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >> LogStreamModule.java:117) >> while locating org.apache.mesos.Log >> at >> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLogInterf >> ace(MesosLogStreamModule.java:152) >> while locating org.apache.aurora.scheduler.log.mesos.LogInterface >> >> 1 error >> com.google.inject.ProvisionException: Guice provision errors: >> >> 1) Error in custom provider, java.lang.IllegalArgumentException: Path >> cannot be >> null >> at >> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >> LogStreamModule.java:117) >> at >> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos >> LogStreamModule.java:117) >> while locating org.apache.mesos.Log >> at >> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLogInterf >> ace(MesosLogStreamModule.java:152) >> while locating org.apache.aurora.scheduler.log.mesos.LogInterface >> >> 1 error >> at >> com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987) >> at >> org.apache.aurora.scheduler.log.mesos.MesosLog.open(MesosLog.java:136 >> >> >> >> -- >> κρισhναν >> >> On Tue, Oct 20, 2015 at 6:14 AM, Bill Farner <[email protected]> wrote: >> >>> The typical flow is that you keep your .aurora file checked into git, >>> and commit every time you deploy/update. When you change your file, you >>> will instruct Aurora to update the live job (have a look at aurora >>> update -h). Aurora will perform a rolling upgrade of your job to the >>> new config. You'll use this same flow for updating your job's software as >>> well as resizing the job. >>> >>> For (3), you could set up alerting for stats that the scheduler >>> exports. Have a look here for monitoring background: >>> https://github.com/apache/aurora/blob/master/docs/monitoring.md >>> >>> You'll find want to look at scheduler stats related to 'pending'. >>> >>> On Mon, Oct 19, 2015 at 12:16 PM, Krish <[email protected]> >>> wrote: >>> >>>> Thanks for the pointer. Now I notice that the aurora-scheduler script >>>> has the --thermos_executor_path as a mandatory requirement. >>>> >>>> I have a couple of questions on how the thermos_executor/.aurora config >>>> file functions: >>>> 1. Do we have to statically define the file beforehand? >>>> 2. What happens when we want to dynamically change the config, say >>>> increasing the number of instances of a service required? Does aurora >>>> require a reboot then? >>>> 3. How do I get notified about the message mesos sends when it cannot >>>> schedule tasks for lack of resources? Should I depend on aurora for this or >>>> try to look for a hook into mesos? >>>> >>>> I think a little bit of context would help here. >>>> What I plan to check is to run a very basic job/task inside a docker >>>> container with aurora & wait for a 'resource not available' message from >>>> mesos, and accordingly call an api to spin up a new node in my cluster. >>>> >>>> >>>> >>>> >>>> -- >>>> κρισhναν >>>> >>>> On Mon, Oct 19, 2015 at 1:24 PM, Erb, Stephan < >>>> [email protected]> wrote: >>>> >>>>> I believe you are missing the thermos_executor options that have to be >>>>> passed to the scheduler command line. >>>>> >>>>> >>>>> See >>>>> https://github.com/apache/aurora/blob/33d7e2170a86f54722a02a2dc9cb1e09fb52df25/examples/vagrant/upstart/aurora-scheduler.conf#L39 >>>>> for an example >>>>> >>>>> >>>>> Best Regards, >>>>> >>>>> Stephan >>>>> >>>>> >>>>> ------------------------------ >>>>> *From:* Krish <[email protected]> >>>>> *Sent:* Monday, October 19, 2015 8:45 AM >>>>> *To:* [email protected] >>>>> *Subject:* Re: Stacktrace when running Apache Aurora >>>>> >>>>> Hi, >>>>> I am a n00b with apache aurora & trying to experiment some things on >>>>> my local machine with zookeeper and mesos-master running locally. They >>>>> have >>>>> initialized properly. When I try to run aurora with the required options, >>>>> I >>>>> get the following error, & googing hasn't helped me much here. >>>>> Appreciate any help. Thanks in advance. >>>>> >>>>> ... >>>>> ... >>>>> WARNING: Method [public void >>>>> org.apache.aurora.scheduler.storage.log.SnapshotStoreImpl.applySnapshot(java.lang.Object)] >>>>> is synthetic and is being intercepted by >>>>> [com.twitter.common.inject.TimedInterceptor@604c5de8]. This could >>>>> indicate a bug. The method >>>>> may be intercepted twice, or may not be intercepted at all. >>>>> Exception in thread "main" com.google.inject.CreationException: Guice >>>>> creation errors: >>>>> >>>>> 1) An exception was caught and reported. Message: A value may only be >>>>> retrieved from a variable that has a default or has been >>>>> set. >>>>> at >>>>> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133) >>>>> >>>>> 2) Could not find a suitable constructor in >>>>> org.apache.aurora.scheduler.mesos.ExecutorSettings. Classes must have >>>>> either one (a >>>>> nd only one) constructor annotated with @Inject or a zero-argument >>>>> constructor that is not private. >>>>> at >>>>> org.apache.aurora.scheduler.mesos.ExecutorSettings.class(ExecutorSettings.java:43) >>>>> at >>>>> org.apache.aurora.scheduler.app.SchedulerMain$3.configure(SchedulerMain.java:204) >>>>> >>>>> 2 errors >>>>> at >>>>> com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) >>>>> at >>>>> com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) >>>>> at >>>>> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) >>>>> at com.google.inject.Guice.createInjector(Guice.java:95) >>>>> at com.google.inject.Guice.createInjector(Guice.java:83) >>>>> at >>>>> com.twitter.common.application.AppLauncher.configureInjection(AppLauncher.java:120) >>>>> at >>>>> com.twitter.common.application.AppLauncher.run(AppLauncher.java:87) >>>>> at >>>>> com.twitter.common.application.AppLauncher.launch(AppLauncher.java:181) >>>>> at >>>>> com.twitter.common.application.AppLauncher.launch(AppLauncher.java:142) >>>>> at >>>>> org.apache.aurora.scheduler.app.SchedulerMain.main(SchedulerMain.java:263) >>>>> Caused by: java.lang.IllegalStateException: A value may only be >>>>> retrieved from a variable that has a default or has been set. >>>>> at >>>>> com.google.common.base.Preconditions.checkState(Preconditions.java:176) >>>>> at com.twitter.common.args.Arg.get(Arg.java:82) >>>>> at >>>>> org.apache.aurora.scheduler.app.SchedulerMain$3.configure(SchedulerMain.java:206) >>>>> at >>>>> com.google.inject.AbstractModule.configure(AbstractModule.java:59) >>>>> at >>>>> com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223) >>>>> at com.google.inject.util.Modules$2.configure(Modules.java:114) >>>>> at >>>>> com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223) >>>>> at >>>>> com.google.inject.spi.Elements.getElements(Elements.java:101) >>>>> at >>>>> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133) >>>>> at >>>>> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103) >>>>> ... 7 more >>>>> >>>>> Complete logs are present @http://pastebin.com/i72HvbYi. >>>>> >>>>> >>>>> >>>>>> -- >>>>>> κρισhναν >>>>>> >>>>> >>>>> >>>> >>> >> >
