Ophir,

It's not possible to have stop/start be the last part of the line, as this
is simply how upstart works. stop/start in this case is actually the name
of the script. They are actually just symlinks to initctl. (That is,
/sbin/{stop,start,etc.} are just symlinks to /sbin/initctl, and
/sbin/initctl can either be called directly with an action as one of the
arguments, or by using one of the symlinks to indicate the action. See "man
initctl" for more information.)

What is your use case for needing to restart the Zeppelin server on EMR so
often?

And yes, it seems that there is a bug in the stop script for Zeppelin. I
think I have fixed it and will include the fix in a future release of EMR.
Sorry for any inconvenience this has caused you! In the meantime you can
kill all processes being run by the zeppelin user, and the Zeppelin server
will automatically start back up in a few seconds (so long as you have not
run "sudo stop zeppelin").

~ Jonathan

On Sun, Jan 17, 2016 at 2:16 AM Ophir Cohen <oph...@gmail.com> wrote:

> Got you.
> OK, good to know that you guys moved to upstart.
> Personally I think that the stop/start command should be the last in the
> line as this is what you mostly change and it is more easy to change.
> In test mode you need to go back to the start/stop all the time and its
> pretty annoying.
>
> Anyway, can it be that the stop command does not work?
> For some reason the start command starts my zeppelin but stop does not,
> well, stop...
>
> On Fri, Jan 15, 2016 at 7:22 PM, Jonathan Kelly <jonathaka...@gmail.com>
> wrote:
>
>> Thanks!
>>
>> I think rather than "sudo /etc/init.d/zeppelin start" the usual way would
>> be "sudo service zeppelin start". However, on EMR 4.x, we actually use
>> upstart (http://upstart.ubuntu.com) to manage the service processes and
>> ensure that they are restarted automatically if they die, so the
>> corresponding command on EMR is "sudo start zeppelin".
>>
>> On Jan 14, 2016, at 11:13 PM, Ophir Cohen <oph...@gmail.com> wrote:
>>
>> Yes,  I just checked that out and find out it does not work on 0.6.
>> Ok, good luck, you are doing great job in emr!
>>
>> BTW
>> If I may say, you need to add service support to Zeppelin.
>> I.e. to allow start of Zeppelin this way:
>> sudo /etc/init.d/zeppelin start
>>
>> *that will start zeppelin under Zeppelin user.
>> On Jan 15, 2016 9:03 AM, "Jonathan Kelly" <jonathaka...@gmail.com> wrote:
>>
>>> Yes, there are a lot of changes since 0.5.5, of course, but we're not
>>> going to support an unreleased version on EMR. We did that for emr-4.1.0
>>> (the first version to support Zeppelin), but it's much less confusing for
>>> us to support only released versions on EMR. As I mentioned earlier, we
>>> will upgrade to 0.5.6 once it's released, but as I also mentioned earlier,
>>> while the Notebook API appears fixed on 0.5.6, the Job API is not. The
>>> Job API, btw, is what allows you to run notebooks via the REST API. The
>>> Notebook API only lets you do CRUD on notebooks/paragraphs/crons.
>>>
>>> FWIW, I also tried building Zeppelin from the tip of the master branch
>>> but still hit the same error when using the Job API, but that's not too
>>> surprising because v0.5.6 isn't too different from the tip of the master
>>> branch, particularly in the REST API code (other than the new Shiro
>>> Security API).
>>>
>>> On Thu, Jan 14, 2016 at 10:45 PM Ophir Cohen <oph...@gmail.com> wrote:
>>>
>>>> I havn't try the job api, in fact what is the differences between
>>>> notebook and job api?
>>>> Anyway, why not jumping to 0.6.0?
>>>> Looks like there is MANY improvments and bug fixes there....
>>>> Sounds to me that you are debugging last year snow 😉
>>>> On Jan 15, 2016 3:07 AM, "Jonathan Kelly" <jonathaka...@gmail.com>
>>>> wrote:
>>>>
>>>>> But now some bad news. I was playing around with the REST API a little
>>>>> more, and I noticed that I get the same AbstractMethodError on Zeppelin
>>>>> 0.5.6 for a POST to /api/job/2A94M5J1Z. :( Does anybody have any idea how
>>>>> this error could possibly have been fixed for /api/notebook in one 
>>>>> release,
>>>>> for /api/notebook/* in the next release, and yet still fail for /api/job/*
>>>>> in that same release?
>>>>>
>>>>> On Thu, Jan 14, 2016 at 4:13 PM Jonathan Kelly <jonathaka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Oh, hey, good news! I just tried Zeppelin 0.5.6 on EMR, and the
>>>>>> Notebook API works fine. So now we just need Zeppelin 0.5.6 to be
>>>>>> officially released, and then we can support it on a future version of 
>>>>>> EMR.
>>>>>>
>>>>>> On Thu, Jan 14, 2016 at 2:01 AM Ophir Cohen <oph...@gmail.com> wrote:
>>>>>>
>>>>>>> OK, I've made it work.
>>>>>>> As Jonathan stated above, on emr 4.1 the notebook APIs does not work
>>>>>>> at all. On both, emr 4.2 and emr 4.3 the /api/notebook works but the 
>>>>>>> query
>>>>>>> for a specific notebook not.
>>>>>>>
>>>>>>> Deploying vanilla Zeppelin (master branch) on the same cluster
>>>>>>> worked as a charm.
>>>>>>>
>>>>>>> It probably origin by Zeppelin versions (0.5.5 on emr vs. 0.6.0 the
>>>>>>> latest).
>>>>>>>
>>>>>>> Now I'm encountering different error that, as far as I can see, does
>>>>>>> not interfere Zeppelin work:
>>>>>>> ERROR [2016-01-14 09:55:38,499] ({qtp1505370540-32}
>>>>>>> NotebookServer.java[onMessage]:176) - Can't handle message
>>>>>>> java.lang.NullPointerException
>>>>>>>
>>>>>>> Many of those.
>>>>>>> If anyone knows what does it mean....
>>>>>>>
>>>>>>> On Thu, Jan 14, 2016 at 12:23 AM, Jonathan Kelly <
>>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Interesting... I confirmed that removing all of the extra classpath
>>>>>>>> entries added by EMR in zeppelin-env.sh does not fix this issue. This 
>>>>>>>> must
>>>>>>>> mean that the issue is caused by the custom EMR builds of either Spark 
>>>>>>>> or
>>>>>>>> Hadoop, which we use when building Zeppelin.
>>>>>>>>
>>>>>>>> On Wed, Jan 13, 2016 at 2:15 PM Jonathan Kelly <
>>>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Ah, never mind. I had incorrectly been excluding the jars in
>>>>>>>>> /usr/lib/zeppelin/interpreter, and I just noticed that 
>>>>>>>>> jersey-core-1.9.jar
>>>>>>>>> appears in two places underneath that directory (in the hive and 
>>>>>>>>> phoenix
>>>>>>>>> interpreter libs), and those jars are included in the Zeppelin server
>>>>>>>>> classpath. This older version of jersey-core conflicts with
>>>>>>>>> jersey-core-1.13.jar from /usr/lib/zeppelin/lib, so this probably 
>>>>>>>>> doesn't
>>>>>>>>> help. However, removing just these two jars did not help by
>>>>>>>>> itself.
>>>>>>>>>
>>>>>>>>> Haven't tried removing classpath entries from zeppelin-env.sh yet
>>>>>>>>> though.
>>>>>>>>>
>>>>>>>>> On Wed, Jan 13, 2016 at 2:07 PM Jonathan Kelly <
>>>>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yeah, I'm sure you're right that it must be due to something that
>>>>>>>>>> EMR is putting in the classpath, but the thing is that I didn't see
>>>>>>>>>> anything else in the classpath that includes the 
>>>>>>>>>> javax.ws.rs.core.Response
>>>>>>>>>> class.
>>>>>>>>>>
>>>>>>>>>> And yes, I agree that trying to remove these two jars from the
>>>>>>>>>> Zeppelin classpath was a futile attempt and would only have helped if
>>>>>>>>>> somehow these jars were near duplicates of each other but different
>>>>>>>>>> versions of the same thing. Besides, if removing one of the jars had
>>>>>>>>>> helped, it would have shown that this was probably not a problem 
>>>>>>>>>> with EMR
>>>>>>>>>> specifically but with Zeppelin itself, which of course would have 
>>>>>>>>>> been
>>>>>>>>>> doubtful.
>>>>>>>>>>
>>>>>>>>>> Anyway, I'll try removing the EMR-supplied extra classpath
>>>>>>>>>> entries from zeppelin-env.sh.
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 13, 2016 at 1:48 PM Ophir Cohen <oph...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Interesting
>>>>>>>>>>>
>>>>>>>>>>> I'm using emr 4.1.0,  I'll try 4.2.0 tomorrow.
>>>>>>>>>>>
>>>>>>>>>>> If I need to guess, I don't think that the mis-version
>>>>>>>>>>> dependency comes from the zeppelin jars as Zeppelin api works when 
>>>>>>>>>>> not in
>>>>>>>>>>> emr.
>>>>>>>>>>> It most likely that it comes from the emr classpath.
>>>>>>>>>>> I would have removed all the additional paths (set in
>>>>>>>>>>> zeppelin-env.sh) and check again.
>>>>>>>>>>>
>>>>>>>>>>> BTW
>>>>>>>>>>> Removing the one of the jars from Zeppelin as you did won't work
>>>>>>>>>>> as Zeppelin needs these jars.
>>>>>>>>>>> If you really want to test it you need to build zeppelin and
>>>>>>>>>>> instructe maven to exclude the problematic dependency  fron one of 
>>>>>>>>>>> these
>>>>>>>>>>> jars.
>>>>>>>>>>> In other words, maven should bring the jar, but without the
>>>>>>>>>>> duplicate dependency.
>>>>>>>>>>> Having said that, and as I stated above, I think that the
>>>>>>>>>>> problematic dependency comes from the additional classpath and not 
>>>>>>>>>>> from
>>>>>>>>>>> Zeppelin core jars.
>>>>>>>>>>> On Jan 13, 2016 11:08 PM, "Jonathan Kelly" <
>>>>>>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, Ophir, (Jonathan from EMR here)
>>>>>>>>>>>>
>>>>>>>>>>>> Are you using emr-4.1.0 or emr-4.2.0? I just tried this with
>>>>>>>>>>>> emr-4.2.0 and found that http://<my-server>:8890/api/notebook
>>>>>>>>>>>> works for me, but http://<my-server>:8890/api/notebook/2A94M5J1Z
>>>>>>>>>>>> fails. For some reason it doesn't quite fail with the same 
>>>>>>>>>>>> exception that
>>>>>>>>>>>> you're seeing, but it definitely seems like a similar cause 
>>>>>>>>>>>> anyway. Here's
>>>>>>>>>>>> the exception I get:
>>>>>>>>>>>>
>>>>>>>>>>>> WARN [2016-01-13 20:34:52,279] ({qtp508683864-37}
>>>>>>>>>>>> ServletHandler.java[doHandle]:590) - Error for 
>>>>>>>>>>>> /api/notebook/2A94M5J1Z
>>>>>>>>>>>> java.lang.AbstractMethodError:
>>>>>>>>>>>> javax.ws.rs.core.Response.getStatusInfo()Ljavax/ws/rs/core/Response$StatusType;
>>>>>>>>>>>> at
>>>>>>>>>>>> javax.ws.rs.WebApplicationException.validate(WebApplicationException.java:186)
>>>>>>>>>>>> at
>>>>>>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>>>>>>>
>>>>>>>>>>>> I found that the Zeppelin classpath includes two different jars
>>>>>>>>>>>> that contain the javax/ws/rs/core/Response class:
>>>>>>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar and
>>>>>>>>>>>> /usr/lib/zeppelin/lib/jersey-core-1.13.jar
>>>>>>>>>>>>
>>>>>>>>>>>> What's weird though is that if I remove either of these from
>>>>>>>>>>>> the classpath, the Zeppelin server fails to start up for different 
>>>>>>>>>>>> reasons. If
>>>>>>>>>>>> I remove /usr/lib/zeppelin/lib/jersey-core-1.13.jar, I get a
>>>>>>>>>>>> ClassNotFoundException for 
>>>>>>>>>>>> com.sun.jersey.core.util.FeaturesAndProperties,
>>>>>>>>>>>> and if I remove /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar,
>>>>>>>>>>>> I get a ClassNotFoundException for
>>>>>>>>>>>> javax.ws.rs.NotFoundException.
>>>>>>>>>>>>
>>>>>>>>>>>> According to a `mvn ... dependency:tree` in the zeppelin-server
>>>>>>>>>>>> submodule, /usr/lib/zeppelin/lib/jersey-core-1.13.jar comes 
>>>>>>>>>>>> indirectly from
>>>>>>>>>>>> zeppelin-server's direct dependency
>>>>>>>>>>>> on com.sun.jersey:jersey-servlet:jar:1.13:compile, and
>>>>>>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar comes from
>>>>>>>>>>>> zeppelin-server's direct dependency on 
>>>>>>>>>>>> javax.ws.rs:javax.ws.rs-api:jar:2.0-m10:compile.
>>>>>>>>>>>> Both of these dependencies seem to have been added a long time 
>>>>>>>>>>>> ago, and
>>>>>>>>>>>> both seem to be used by the websocket API rather than the REST API 
>>>>>>>>>>>> (at
>>>>>>>>>>>> least based upon the Git commit descriptions where they were 
>>>>>>>>>>>> added), so
>>>>>>>>>>>> this confuses me even more.
>>>>>>>>>>>>
>>>>>>>>>>>> Does anybody from the Zeppelin side have any idea what is going
>>>>>>>>>>>> on here?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Jonathan
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 13, 2016 at 5:34 AM Ophir Cohen <oph...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>> We migrated our Zeppelin to use EMR Zeppelin. It's straight
>>>>>>>>>>>>> forward and we were happy with the migration till we found out 
>>>>>>>>>>>>> that
>>>>>>>>>>>>> something isn't working well with the rest API.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Calling to the interpreter API:
>>>>>>>>>>>>> http://<my-server>:8890/api/interpreter
>>>>>>>>>>>>> and this:
>>>>>>>>>>>>> http://<my-server>:8890/api/interpreter/setting
>>>>>>>>>>>>> Returns results as expected.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When trying to access the notebooks:
>>>>>>>>>>>>> http://<my-server>:8890/api/notebook
>>>>>>>>>>>>> Or a specific notebook:
>>>>>>>>>>>>> http://<my-server>:8890/api/notebook/2A94M5J1Z
>>>>>>>>>>>>> It fails with HTTP 500 error.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The error I can see in the logs is NoSuchMethodError (see
>>>>>>>>>>>>> below) which suggests we might have here 'jar-hell' and the wrong 
>>>>>>>>>>>>> (probably
>>>>>>>>>>>>> old) jar loaded instead of the need one - but I can't figure that 
>>>>>>>>>>>>> out.
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> The exception:
>>>>>>>>>>>>> WARN [2016-01-13 07:47:11,690] ({qtp716961517-38}
>>>>>>>>>>>>> ServletHandler.java[doHandle]:590) - Error for 
>>>>>>>>>>>>> /api/notebook/2A94M5J1Z
>>>>>>>>>>>>> java.lang.NoSuchMethodError:
>>>>>>>>>>>>> javax.ws.rs.ClientErrorException.validate(Ljavax/ws/rs/core/Response;Ljavax/ws/rs/core/Response$Status$Family;)Ljavax/ws/rs/core/Response;
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>
>

Reply via email to