Oh, hey, good news! I just tried Zeppelin 0.5.6 on EMR, and the Notebook
API works fine. So now we just need Zeppelin 0.5.6 to be officially
released, and then we can support it on a future version of EMR.
On Thu, Jan 14, 2016 at 2:01 AM Ophir Cohen <oph...@gmail.com> wrote:
> OK, I've made it work.
> As Jonathan stated above, on emr 4.1 the notebook APIs does not work at
> all. On both, emr 4.2 and emr 4.3 the /api/notebook works but the query for
> a specific notebook not.
>
> Deploying vanilla Zeppelin (master branch) on the same cluster worked as a
> charm.
>
> It probably origin by Zeppelin versions (0.5.5 on emr vs. 0.6.0 the
> latest).
>
> Now I'm encountering different error that, as far as I can see, does not
> interfere Zeppelin work:
> ERROR [2016-01-14 09:55:38,499] ({qtp1505370540-32}
> NotebookServer.java[onMessage]:176) - Can't handle message
> java.lang.NullPointerException
>
> Many of those.
> If anyone knows what does it mean....
>
> On Thu, Jan 14, 2016 at 12:23 AM, Jonathan Kelly <jonathaka...@gmail.com>
> wrote:
>
>> Interesting... I confirmed that removing all of the extra classpath
>> entries added by EMR in zeppelin-env.sh does not fix this issue. This must
>> mean that the issue is caused by the custom EMR builds of either Spark or
>> Hadoop, which we use when building Zeppelin.
>>
>> On Wed, Jan 13, 2016 at 2:15 PM Jonathan Kelly <jonathaka...@gmail.com>
>> wrote:
>>
>>> Ah, never mind. I had incorrectly been excluding the jars in
>>> /usr/lib/zeppelin/interpreter, and I just noticed that jersey-core-1.9.jar
>>> appears in two places underneath that directory (in the hive and phoenix
>>> interpreter libs), and those jars are included in the Zeppelin server
>>> classpath. This older version of jersey-core conflicts with
>>> jersey-core-1.13.jar from /usr/lib/zeppelin/lib, so this probably doesn't
>>> help. However, removing just these two jars did not help by itself.
>>>
>>> Haven't tried removing classpath entries from zeppelin-env.sh yet though.
>>>
>>> On Wed, Jan 13, 2016 at 2:07 PM Jonathan Kelly <jonathaka...@gmail.com>
>>> wrote:
>>>
>>>> Yeah, I'm sure you're right that it must be due to something that EMR
>>>> is putting in the classpath, but the thing is that I didn't see anything
>>>> else in the classpath that includes the javax.ws.rs.core.Response class.
>>>>
>>>> And yes, I agree that trying to remove these two jars from the Zeppelin
>>>> classpath was a futile attempt and would only have helped if somehow these
>>>> jars were near duplicates of each other but different versions of the same
>>>> thing. Besides, if removing one of the jars had helped, it would have shown
>>>> that this was probably not a problem with EMR specifically but with
>>>> Zeppelin itself, which of course would have been doubtful.
>>>>
>>>> Anyway, I'll try removing the EMR-supplied extra classpath entries from
>>>> zeppelin-env.sh.
>>>>
>>>> On Wed, Jan 13, 2016 at 1:48 PM Ophir Cohen <oph...@gmail.com> wrote:
>>>>
>>>>> Interesting
>>>>>
>>>>> I'm using emr 4.1.0, I'll try 4.2.0 tomorrow.
>>>>>
>>>>> If I need to guess, I don't think that the mis-version dependency
>>>>> comes from the zeppelin jars as Zeppelin api works when not in emr.
>>>>> It most likely that it comes from the emr classpath.
>>>>> I would have removed all the additional paths (set in zeppelin-env.sh)
>>>>> and check again.
>>>>>
>>>>> BTW
>>>>> Removing the one of the jars from Zeppelin as you did won't work as
>>>>> Zeppelin needs these jars.
>>>>> If you really want to test it you need to build zeppelin and instructe
>>>>> maven to exclude the problematic dependency fron one of these jars.
>>>>> In other words, maven should bring the jar, but without the duplicate
>>>>> dependency.
>>>>> Having said that, and as I stated above, I think that the problematic
>>>>> dependency comes from the additional classpath and not from Zeppelin core
>>>>> jars.
>>>>> On Jan 13, 2016 11:08 PM, "Jonathan Kelly" <jonathaka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi, Ophir, (Jonathan from EMR here)
>>>>>>
>>>>>> Are you using emr-4.1.0 or emr-4.2.0? I just tried this with
>>>>>> emr-4.2.0 and found that http://<my-server>:8890/api/notebook works
>>>>>> for me, but http://<my-server>:8890/api/notebook/2A94M5J1Z fails.
>>>>>> For some reason it doesn't quite fail with the same exception that you're
>>>>>> seeing, but it definitely seems like a similar cause anyway. Here's the
>>>>>> exception I get:
>>>>>>
>>>>>> WARN [2016-01-13 20:34:52,279] ({qtp508683864-37}
>>>>>> ServletHandler.java[doHandle]:590) - Error for /api/notebook/2A94M5J1Z
>>>>>> java.lang.AbstractMethodError:
>>>>>> javax.ws.rs.core.Response.getStatusInfo()Ljavax/ws/rs/core/Response$StatusType;
>>>>>> at
>>>>>> javax.ws.rs.WebApplicationException.validate(WebApplicationException.java:186)
>>>>>> at
>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>> at
>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>> at
>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>
>>>>>> I found that the Zeppelin classpath includes two different jars that
>>>>>> contain the javax/ws/rs/core/Response class:
>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar
>>>>>> and /usr/lib/zeppelin/lib/jersey-core-1.13.jar
>>>>>>
>>>>>> What's weird though is that if I remove either of these from the
>>>>>> classpath, the Zeppelin server fails to start up for different reasons.
>>>>>> If
>>>>>> I remove /usr/lib/zeppelin/lib/jersey-core-1.13.jar, I get a
>>>>>> ClassNotFoundException for
>>>>>> com.sun.jersey.core.util.FeaturesAndProperties,
>>>>>> and if I remove /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar, I
>>>>>> get a ClassNotFoundException for javax.ws.rs.NotFoundException.
>>>>>>
>>>>>> According to a `mvn ... dependency:tree` in the zeppelin-server
>>>>>> submodule, /usr/lib/zeppelin/lib/jersey-core-1.13.jar comes indirectly
>>>>>> from
>>>>>> zeppelin-server's direct dependency
>>>>>> on com.sun.jersey:jersey-servlet:jar:1.13:compile, and
>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar
>>>>>> comes from zeppelin-server's direct dependency on
>>>>>> javax.ws.rs:javax.ws.rs-api:jar:2.0-m10:compile.
>>>>>> Both of these dependencies seem to have been added a long time ago, and
>>>>>> both seem to be used by the websocket API rather than the REST API (at
>>>>>> least based upon the Git commit descriptions where they were added), so
>>>>>> this confuses me even more.
>>>>>>
>>>>>> Does anybody from the Zeppelin side have any idea what is going on
>>>>>> here?
>>>>>>
>>>>>> Thanks,
>>>>>> Jonathan
>>>>>>
>>>>>> On Wed, Jan 13, 2016 at 5:34 AM Ophir Cohen <oph...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>> We migrated our Zeppelin to use EMR Zeppelin. It's straight forward
>>>>>>> and we were happy with the migration till we found out that something
>>>>>>> isn't
>>>>>>> working well with the rest API.
>>>>>>>
>>>>>>> Calling to the interpreter API:
>>>>>>> http://<my-server>:8890/api/interpreter
>>>>>>> and this:
>>>>>>> http://<my-server>:8890/api/interpreter/setting
>>>>>>> Returns results as expected.
>>>>>>>
>>>>>>> When trying to access the notebooks:
>>>>>>> http://<my-server>:8890/api/notebook
>>>>>>> Or a specific notebook:
>>>>>>> http://<my-server>:8890/api/notebook/2A94M5J1Z
>>>>>>> It fails with HTTP 500 error.
>>>>>>>
>>>>>>> The error I can see in the logs is NoSuchMethodError (see below)
>>>>>>> which suggests we might have here 'jar-hell' and the wrong (probably
>>>>>>> old)
>>>>>>> jar loaded instead of the need one - but I can't figure that out.
>>>>>>> Thanks!
>>>>>>>
>>>>>>> The exception:
>>>>>>> WARN [2016-01-13 07:47:11,690] ({qtp716961517-38}
>>>>>>> ServletHandler.java[doHandle]:590) - Error for /api/notebook/2A94M5J1Z
>>>>>>> java.lang.NoSuchMethodError:
>>>>>>> javax.ws.rs.ClientErrorException.validate(Ljavax/ws/rs/core/Response;Ljavax/ws/rs/core/Response$Status$Family;)Ljavax/ws/rs/core/Response;
>>>>>>> at
>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>> at
>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>> at
>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>>
>>>>>>
>
