Thanks, Zameer.
I had to modify /etc/aurora/clusters.json:
[
{
"auth_mechanism": "UNAUTHENTICATED",
"name": "testcluster",
"scheduler_zk_path": "/scheduler/aurora",
"slave_root": "/var/lib/mesos",
"slave_run_directory": "latest",
"zk": "127.0.1.1"
}
]
I have a hello_world.aurora in my home folder. However the following
command errors out:
~$ aurora job create testcluster/testrole/test/hellojob
./hello_world.aurora
Error loading configuration: [Errno 2] No such file or directory:
'/vagrant/hello_world.py'
A job list does work:
~$ aurora job list testcluster
INFO] Retrieving jobs for role None
I am not even using the vagrant. I am using zk & mesos on the same machine
as aurora. How do I submit these job templates to aurora?
Any pointers to documentation will be helpful.
--
κρισhναν
On Wed, Oct 21, 2015 at 11:09 PM, Zameer Manji <[email protected]> wrote:
> Version 0.9.0 does not have the gc executor. Version 0.9.0 uses Mesos' task
> reconciliation
> <http://mesos.apache.org/documentation/latest/reconciliation/> API
> instead.
>
> On Wed, Oct 21, 2015 at 9:28 AM, Krish <[email protected]> wrote:
>
>> Thanks Bill for the location to the debs. I was finally able to run
>> aurora. :)
>>
>> I did find thermos_executor.pex & thermos_observer after installing
>> aurora-executor. I still could not find gc_executor.pex on my system.
>> Is there a location from where I can download the binaries for *.pex or
>> build them from scratch?
>>
>> root@dev:/# find . -name "*.pex"
>> ./usr/share/aurora/bin/thermos_executor.pex
>> ./usr/share/aurora/bin/kaurora_admin.pex
>> ./usr/share/aurora/bin/kaurora.pex
>> ./usr/share/aurora/bin/thermos.pex
>> ./usr/share/aurora/bin/thermos_observer.pex
>> ./home/ubuntu/.pex
>> ./root/.pex
>>
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Tue, Oct 20, 2015 at 11:46 PM, Bill Farner <[email protected]> wrote:
>>
>>> Aurora currently requires an executor, so setting it to /dev/null will
>>> not work. Happy to talk further about your thoughts around sidestepping
>>> the executor.
>>>
>>> As for working with the scheduler source code, it's a standard gradle
>>> project and we tend to use intellij. Docs to help ramp on that:
>>> https://github.com/apache/aurora/blob/master/docs/developing-aurora-scheduler.md
>>>
>>> As for builds - the .zip is a source distribution, so it won't have any
>>> pre-built binaries. If you're on debian, we have official debs here:
>>> https://bintray.com/apache/aurora
>>> You can see how they're built here (and can build your own) packages:
>>> https://github.com/apache/aurora-packaging
>>> We're close to having official RPMs, but none to speak of yet.
>>>
>>> On Tue, Oct 20, 2015 at 9:47 AM, Krish <[email protected]>
>>> wrote:
>>>
>>>> Stephen,
>>>> I am trying to get started and run aurora without thermos executor
>>>> (setting it to /dev/null does not help) - on a local linux box for now &
>>>> planning to containerize/dockerize it later.
>>>>
>>>> Can you please point me to the right documentation (or a pointer to the
>>>> cli parsing source code) which can help me resolve this? Also, are there
>>>> any steps steps to import source code into eclipse to browse & analyze code
>>>> for this.
>>>>
>>>> Also, where do i find all the *.pex files? They are not present in the
>>>> zip file nor anywhere in the built source code.
>>>>
>>>> I know I am asking too many queries on a single thread here, & would
>>>> appreciate the help.
>>>> I think at the end of this, I will put the steps I followed in a
>>>> gist/blog so others might find their way around, & not struggle as much.
>>>>
>>>>
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Tue, Oct 20, 2015 at 4:09 PM, Erb, Stephan <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Krish,
>>>>>
>>>>>
>>>>> you don't have to set framework_authentication_file and
>>>>> zk_digest_credentials. The scheduler help text is misleading here as
>>>>> everything will work fine if you leave those empty.
>>>>>
>>>>>
>>>>> In addition, looks like you are misunderstanding the usage of the
>>>>> thermos_executor_path command line flag of the scheduler. It is
>>>>> supposed to point to the binary containing the generic Aurora executor
>>>>> (thermos_executor.pex). You only need the hello_world.aurora once
>>>>> your scheduler is up an running. It serves as an example input for the
>>>>> aurora command line client which can be used to scheduler jobs and
>>>>> services
>>>>> on an Aurora master.
>>>>>
>>>>>
>>>>> Have you tried to use the vagrant box? Just type 'vagrant up`in a
>>>>> checkout of the Aurora source code. It gives you a running scheduler to
>>>>> play with. Once you have understood how it works, you can start trying to
>>>>> install it on your own (by reverse-engineering the vagrant box).
>>>>>
>>>>>
>>>>> Hope this helps a little,
>>>>>
>>>>> Stephan
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> *From:* Krish <[email protected]>
>>>>> *Sent:* Tuesday, October 20, 2015 11:39 AM
>>>>> *To:* Bill Farner
>>>>> *Cc:* [email protected]; Erb, Stephan
>>>>>
>>>>> *Subject:* Re: Stacktrace when running Apache Aurora
>>>>>
>>>>> Bill/Stephen,
>>>>> I still get a stacktrace when running the aurora scheduler CLI.
>>>>>
>>>>> I do not know what to specify for -framework_authentication_file
>>>>> & -zk_digest_credentials, and they are required arguments.
>>>>>
>>>>> I am not using any authentication on Mesos master, do I still need the
>>>>> framework_authentication_file parameter?
>>>>>
>>>>>
>>>>> rm -rf /db /backup_dir
>>>>> mesos-log initialize --path="/db"
>>>>> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
>>>>> JAVA_OPTS="-Xmx1536m -Xms256m"
>>>>> /usr/local/aurora-scheduler/bin/aurora-scheduler -backup_dir=/backup_dir
>>>>> -cluster_name=tc -mesos_master_address=zk://localhost:2181/mesos/master
>>>>> -serverset_path=/scheduler/aurora -zk_endpoints=localhost:2181
>>>>> -native_log_quorum_size=1 -vlog=SEVERE -logtostderr=false
>>>>> -native_log_file_path=/db
>>>>> -thermos_executor_path=/home/ubuntu/hello_world.aurora
>>>>> ...
>>>>> ...
>>>>> INFO: Binding org.apache.aurora.scheduler.http.Utilization to
>>>>> GuiceManagedCompon
>>>>> entProvider with the scope "PerRequest"
>>>>> Oct 20, 2015 9:27:40 AM
>>>>> org.apache.aurora.scheduler.cron.quartz.CronModule provi
>>>>> deTimeZone
>>>>> WARNING: Cron schedules are configured to fire according to timezone
>>>>> Greenwich M
>>>>> ean Time but system timezone is set to Coordinated Universal Time
>>>>> Oct 20, 2015 9:27:41 AM org.eclipse.jetty.server.AbstractConnector
>>>>> doStart
>>>>> INFO: Started [email protected]:43843
>>>>> E1020 09:27:41.290 THREAD1
>>>>> org.apache.aurora.scheduler.SchedulerLifecycle$9.exec
>>>>> ute: Caught unchecked exception: com.google.inject.ProvisionException:
>>>>> Guice pro
>>>>> vision errors:
>>>>>
>>>>> 1) Error in custom provider, java.lang.IllegalArgumentException: Path
>>>>> cannot be null at
>>>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos
>>>>> LogStreamModule.java:117)
>>>>> at
>>>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos
>>>>> LogStreamModule.java:117)
>>>>> while locating org.apache.mesos.Log
>>>>> at
>>>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLogInterf
>>>>> ace(MesosLogStreamModule.java:152)
>>>>> while locating org.apache.aurora.scheduler.log.mesos.LogInterface
>>>>>
>>>>> 1 error
>>>>> com.google.inject.ProvisionException: Guice provision errors:
>>>>>
>>>>> 1) Error in custom provider, java.lang.IllegalArgumentException: Path
>>>>> cannot be
>>>>> null
>>>>> at
>>>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos
>>>>> LogStreamModule.java:117)
>>>>> at
>>>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLog(Mesos
>>>>> LogStreamModule.java:117)
>>>>> while locating org.apache.mesos.Log
>>>>> at
>>>>> org.apache.aurora.scheduler.log.mesos.MesosLogStreamModule.provideLogInterf
>>>>> ace(MesosLogStreamModule.java:152)
>>>>> while locating org.apache.aurora.scheduler.log.mesos.LogInterface
>>>>>
>>>>> 1 error
>>>>> at
>>>>> com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
>>>>> at
>>>>> org.apache.aurora.scheduler.log.mesos.MesosLog.open(MesosLog.java:136
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> κρισhναν
>>>>>
>>>>> On Tue, Oct 20, 2015 at 6:14 AM, Bill Farner <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> The typical flow is that you keep your .aurora file checked into git,
>>>>>> and commit every time you deploy/update. When you change your file, you
>>>>>> will instruct Aurora to update the live job (have a look at aurora
>>>>>> update -h). Aurora will perform a rolling upgrade of your job to
>>>>>> the new config. You'll use this same flow for updating your job's
>>>>>> software
>>>>>> as well as resizing the job.
>>>>>>
>>>>>> For (3), you could set up alerting for stats that the scheduler
>>>>>> exports. Have a look here for monitoring background:
>>>>>> https://github.com/apache/aurora/blob/master/docs/monitoring.md
>>>>>>
>>>>>> You'll find want to look at scheduler stats related to 'pending'.
>>>>>>
>>>>>> On Mon, Oct 19, 2015 at 12:16 PM, Krish <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the pointer. Now I notice that the aurora-scheduler
>>>>>>> script has the --thermos_executor_path as a mandatory requirement.
>>>>>>>
>>>>>>> I have a couple of questions on how the thermos_executor/.aurora
>>>>>>> config file functions:
>>>>>>> 1. Do we have to statically define the file beforehand?
>>>>>>> 2. What happens when we want to dynamically change the config, say
>>>>>>> increasing the number of instances of a service required? Does aurora
>>>>>>> require a reboot then?
>>>>>>> 3. How do I get notified about the message mesos sends when it
>>>>>>> cannot schedule tasks for lack of resources? Should I depend on aurora
>>>>>>> for
>>>>>>> this or try to look for a hook into mesos?
>>>>>>>
>>>>>>> I think a little bit of context would help here.
>>>>>>> What I plan to check is to run a very basic job/task inside a docker
>>>>>>> container with aurora & wait for a 'resource not available' message from
>>>>>>> mesos, and accordingly call an api to spin up a new node in my cluster.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> κρισhναν
>>>>>>>
>>>>>>> On Mon, Oct 19, 2015 at 1:24 PM, Erb, Stephan <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> I believe you are missing the thermos_executor options that have to
>>>>>>>> be passed to the scheduler command line.
>>>>>>>>
>>>>>>>>
>>>>>>>> See
>>>>>>>> https://github.com/apache/aurora/blob/33d7e2170a86f54722a02a2dc9cb1e09fb52df25/examples/vagrant/upstart/aurora-scheduler.conf#L39
>>>>>>>> for an example
>>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>>
>>>>>>>> Stephan
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> *From:* Krish <[email protected]>
>>>>>>>> *Sent:* Monday, October 19, 2015 8:45 AM
>>>>>>>> *To:* [email protected]
>>>>>>>> *Subject:* Re: Stacktrace when running Apache Aurora
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> I am a n00b with apache aurora & trying to experiment some things
>>>>>>>> on my local machine with zookeeper and mesos-master running locally.
>>>>>>>> They
>>>>>>>> have initialized properly. When I try to run aurora with the required
>>>>>>>> options, I get the following error, & googing hasn't helped me much
>>>>>>>> here.
>>>>>>>> Appreciate any help. Thanks in advance.
>>>>>>>>
>>>>>>>> ...
>>>>>>>> ...
>>>>>>>> WARNING: Method [public void
>>>>>>>> org.apache.aurora.scheduler.storage.log.SnapshotStoreImpl.applySnapshot(java.lang.Object)]
>>>>>>>> is synthetic and is being intercepted by
>>>>>>>> [com.twitter.common.inject.TimedInterceptor@604c5de8]. This could
>>>>>>>> indicate a bug. The method
>>>>>>>> may be intercepted twice, or may not be intercepted at all.
>>>>>>>> Exception in thread "main" com.google.inject.CreationException:
>>>>>>>> Guice creation errors:
>>>>>>>>
>>>>>>>> 1) An exception was caught and reported. Message: A value may only
>>>>>>>> be retrieved from a variable that has a default or has been
>>>>>>>> set.
>>>>>>>> at
>>>>>>>> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133)
>>>>>>>>
>>>>>>>> 2) Could not find a suitable constructor in
>>>>>>>> org.apache.aurora.scheduler.mesos.ExecutorSettings. Classes must have
>>>>>>>> either one (a
>>>>>>>> nd only one) constructor annotated with @Inject or a zero-argument
>>>>>>>> constructor that is not private.
>>>>>>>> at
>>>>>>>> org.apache.aurora.scheduler.mesos.ExecutorSettings.class(ExecutorSettings.java:43)
>>>>>>>> at
>>>>>>>> org.apache.aurora.scheduler.app.SchedulerMain$3.configure(SchedulerMain.java:204)
>>>>>>>>
>>>>>>>> 2 errors
>>>>>>>> at
>>>>>>>> com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
>>>>>>>> at
>>>>>>>> com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
>>>>>>>> at
>>>>>>>> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
>>>>>>>> at com.google.inject.Guice.createInjector(Guice.java:95)
>>>>>>>> at com.google.inject.Guice.createInjector(Guice.java:83)
>>>>>>>> at
>>>>>>>> com.twitter.common.application.AppLauncher.configureInjection(AppLauncher.java:120)
>>>>>>>> at
>>>>>>>> com.twitter.common.application.AppLauncher.run(AppLauncher.java:87)
>>>>>>>> at
>>>>>>>> com.twitter.common.application.AppLauncher.launch(AppLauncher.java:181)
>>>>>>>> at
>>>>>>>> com.twitter.common.application.AppLauncher.launch(AppLauncher.java:142)
>>>>>>>> at
>>>>>>>> org.apache.aurora.scheduler.app.SchedulerMain.main(SchedulerMain.java:263)
>>>>>>>> Caused by: java.lang.IllegalStateException: A value may only be
>>>>>>>> retrieved from a variable that has a default or has been set.
>>>>>>>> at
>>>>>>>> com.google.common.base.Preconditions.checkState(Preconditions.java:176)
>>>>>>>> at com.twitter.common.args.Arg.get(Arg.java:82)
>>>>>>>> at
>>>>>>>> org.apache.aurora.scheduler.app.SchedulerMain$3.configure(SchedulerMain.java:206)
>>>>>>>> at
>>>>>>>> com.google.inject.AbstractModule.configure(AbstractModule.java:59)
>>>>>>>> at
>>>>>>>> com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
>>>>>>>> at
>>>>>>>> com.google.inject.util.Modules$2.configure(Modules.java:114)
>>>>>>>> at
>>>>>>>> com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
>>>>>>>> at
>>>>>>>> com.google.inject.spi.Elements.getElements(Elements.java:101)
>>>>>>>> at
>>>>>>>> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133)
>>>>>>>> at
>>>>>>>> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103)
>>>>>>>> ... 7 more
>>>>>>>>
>>>>>>>> Complete logs are present @http://pastebin.com/i72HvbYi.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> --
>>>>>>>>> κρισhναν
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> Zameer Manji
>
