But now some bad news. I was playing around with the REST API a little
more, and I noticed that I get the same AbstractMethodError on Zeppelin
0.5.6 for a POST to /api/job/2A94M5J1Z. :( Does anybody have any idea how
this error could possibly have been fixed for /api/notebook in one release,
for /api/notebook/* in the next release, and yet still fail for /api/job/*
in that same release?
On Thu, Jan 14, 2016 at 4:13 PM Jonathan Kelly <jonathaka...@gmail.com>
wrote:
> Oh, hey, good news! I just tried Zeppelin 0.5.6 on EMR, and the Notebook
> API works fine. So now we just need Zeppelin 0.5.6 to be officially
> released, and then we can support it on a future version of EMR.
>
> On Thu, Jan 14, 2016 at 2:01 AM Ophir Cohen <oph...@gmail.com> wrote:
>
>> OK, I've made it work.
>> As Jonathan stated above, on emr 4.1 the notebook APIs does not work at
>> all. On both, emr 4.2 and emr 4.3 the /api/notebook works but the query for
>> a specific notebook not.
>>
>> Deploying vanilla Zeppelin (master branch) on the same cluster worked as
>> a charm.
>>
>> It probably origin by Zeppelin versions (0.5.5 on emr vs. 0.6.0 the
>> latest).
>>
>> Now I'm encountering different error that, as far as I can see, does not
>> interfere Zeppelin work:
>> ERROR [2016-01-14 09:55:38,499] ({qtp1505370540-32}
>> NotebookServer.java[onMessage]:176) - Can't handle message
>> java.lang.NullPointerException
>>
>> Many of those.
>> If anyone knows what does it mean....
>>
>> On Thu, Jan 14, 2016 at 12:23 AM, Jonathan Kelly <jonathaka...@gmail.com>
>> wrote:
>>
>>> Interesting... I confirmed that removing all of the extra classpath
>>> entries added by EMR in zeppelin-env.sh does not fix this issue. This must
>>> mean that the issue is caused by the custom EMR builds of either Spark or
>>> Hadoop, which we use when building Zeppelin.
>>>
>>> On Wed, Jan 13, 2016 at 2:15 PM Jonathan Kelly <jonathaka...@gmail.com>
>>> wrote:
>>>
>>>> Ah, never mind. I had incorrectly been excluding the jars in
>>>> /usr/lib/zeppelin/interpreter, and I just noticed that jersey-core-1.9.jar
>>>> appears in two places underneath that directory (in the hive and phoenix
>>>> interpreter libs), and those jars are included in the Zeppelin server
>>>> classpath. This older version of jersey-core conflicts with
>>>> jersey-core-1.13.jar from /usr/lib/zeppelin/lib, so this probably doesn't
>>>> help. However, removing just these two jars did not help by itself.
>>>>
>>>> Haven't tried removing classpath entries from zeppelin-env.sh yet
>>>> though.
>>>>
>>>> On Wed, Jan 13, 2016 at 2:07 PM Jonathan Kelly <jonathaka...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yeah, I'm sure you're right that it must be due to something that EMR
>>>>> is putting in the classpath, but the thing is that I didn't see anything
>>>>> else in the classpath that includes the javax.ws.rs.core.Response class.
>>>>>
>>>>> And yes, I agree that trying to remove these two jars from the
>>>>> Zeppelin classpath was a futile attempt and would only have helped if
>>>>> somehow these jars were near duplicates of each other but different
>>>>> versions of the same thing. Besides, if removing one of the jars had
>>>>> helped, it would have shown that this was probably not a problem with EMR
>>>>> specifically but with Zeppelin itself, which of course would have been
>>>>> doubtful.
>>>>>
>>>>> Anyway, I'll try removing the EMR-supplied extra classpath entries
>>>>> from zeppelin-env.sh.
>>>>>
>>>>> On Wed, Jan 13, 2016 at 1:48 PM Ophir Cohen <oph...@gmail.com> wrote:
>>>>>
>>>>>> Interesting
>>>>>>
>>>>>> I'm using emr 4.1.0, I'll try 4.2.0 tomorrow.
>>>>>>
>>>>>> If I need to guess, I don't think that the mis-version dependency
>>>>>> comes from the zeppelin jars as Zeppelin api works when not in emr.
>>>>>> It most likely that it comes from the emr classpath.
>>>>>> I would have removed all the additional paths (set in
>>>>>> zeppelin-env.sh) and check again.
>>>>>>
>>>>>> BTW
>>>>>> Removing the one of the jars from Zeppelin as you did won't work as
>>>>>> Zeppelin needs these jars.
>>>>>> If you really want to test it you need to build zeppelin and
>>>>>> instructe maven to exclude the problematic dependency fron one of these
>>>>>> jars.
>>>>>> In other words, maven should bring the jar, but without the duplicate
>>>>>> dependency.
>>>>>> Having said that, and as I stated above, I think that the problematic
>>>>>> dependency comes from the additional classpath and not from Zeppelin core
>>>>>> jars.
>>>>>> On Jan 13, 2016 11:08 PM, "Jonathan Kelly" <jonathaka...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, Ophir, (Jonathan from EMR here)
>>>>>>>
>>>>>>> Are you using emr-4.1.0 or emr-4.2.0? I just tried this with
>>>>>>> emr-4.2.0 and found that http://<my-server>:8890/api/notebook works
>>>>>>> for me, but http://<my-server>:8890/api/notebook/2A94M5J1Z fails.
>>>>>>> For some reason it doesn't quite fail with the same exception that
>>>>>>> you're
>>>>>>> seeing, but it definitely seems like a similar cause anyway. Here's the
>>>>>>> exception I get:
>>>>>>>
>>>>>>> WARN [2016-01-13 20:34:52,279] ({qtp508683864-37}
>>>>>>> ServletHandler.java[doHandle]:590) - Error for /api/notebook/2A94M5J1Z
>>>>>>> java.lang.AbstractMethodError:
>>>>>>> javax.ws.rs.core.Response.getStatusInfo()Ljavax/ws/rs/core/Response$StatusType;
>>>>>>> at
>>>>>>> javax.ws.rs.WebApplicationException.validate(WebApplicationException.java:186)
>>>>>>> at
>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>> at
>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>> at
>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>>
>>>>>>> I found that the Zeppelin classpath includes two different jars that
>>>>>>> contain the javax/ws/rs/core/Response class:
>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar
>>>>>>> and /usr/lib/zeppelin/lib/jersey-core-1.13.jar
>>>>>>>
>>>>>>> What's weird though is that if I remove either of these from the
>>>>>>> classpath, the Zeppelin server fails to start up for different reasons.
>>>>>>> If
>>>>>>> I remove /usr/lib/zeppelin/lib/jersey-core-1.13.jar, I get a
>>>>>>> ClassNotFoundException for
>>>>>>> com.sun.jersey.core.util.FeaturesAndProperties,
>>>>>>> and if I remove /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar,
>>>>>>> I get a ClassNotFoundException for javax.ws.rs.NotFoundException.
>>>>>>>
>>>>>>> According to a `mvn ... dependency:tree` in the zeppelin-server
>>>>>>> submodule, /usr/lib/zeppelin/lib/jersey-core-1.13.jar comes indirectly
>>>>>>> from
>>>>>>> zeppelin-server's direct dependency
>>>>>>> on com.sun.jersey:jersey-servlet:jar:1.13:compile, and
>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar
>>>>>>> comes from zeppelin-server's direct dependency on
>>>>>>> javax.ws.rs:javax.ws.rs-api:jar:2.0-m10:compile.
>>>>>>> Both of these dependencies seem to have been added a long time ago, and
>>>>>>> both seem to be used by the websocket API rather than the REST API (at
>>>>>>> least based upon the Git commit descriptions where they were added), so
>>>>>>> this confuses me even more.
>>>>>>>
>>>>>>> Does anybody from the Zeppelin side have any idea what is going on
>>>>>>> here?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jonathan
>>>>>>>
>>>>>>> On Wed, Jan 13, 2016 at 5:34 AM Ophir Cohen <oph...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>> We migrated our Zeppelin to use EMR Zeppelin. It's straight forward
>>>>>>>> and we were happy with the migration till we found out that something
>>>>>>>> isn't
>>>>>>>> working well with the rest API.
>>>>>>>>
>>>>>>>> Calling to the interpreter API:
>>>>>>>> http://<my-server>:8890/api/interpreter
>>>>>>>> and this:
>>>>>>>> http://<my-server>:8890/api/interpreter/setting
>>>>>>>> Returns results as expected.
>>>>>>>>
>>>>>>>> When trying to access the notebooks:
>>>>>>>> http://<my-server>:8890/api/notebook
>>>>>>>> Or a specific notebook:
>>>>>>>> http://<my-server>:8890/api/notebook/2A94M5J1Z
>>>>>>>> It fails with HTTP 500 error.
>>>>>>>>
>>>>>>>> The error I can see in the logs is NoSuchMethodError (see below)
>>>>>>>> which suggests we might have here 'jar-hell' and the wrong (probably
>>>>>>>> old)
>>>>>>>> jar loaded instead of the need one - but I can't figure that out.
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> The exception:
>>>>>>>> WARN [2016-01-13 07:47:11,690] ({qtp716961517-38}
>>>>>>>> ServletHandler.java[doHandle]:590) - Error for /api/notebook/2A94M5J1Z
>>>>>>>> java.lang.NoSuchMethodError:
>>>>>>>> javax.ws.rs.ClientErrorException.validate(Ljavax/ws/rs/core/Response;Ljavax/ws/rs/core/Response$Status$Family;)Ljavax/ws/rs/core/Response;
>>>>>>>> at
>>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>>> at
>>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>>> at
>>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>>>
>>>>>>>
>>
