Re: Support for Livy with Scala 2.12
Looks like the issue might be Spark 2.4.2 only? From https://spark.apache.org/downloads.html, "Note that, Spark is pre-built with Scala 2.11 except version 2.4.2, which is pre-built with Scala 2.12." So maybe you just got unlucky with using Spark 2.4.2? Kevin Risden On Mon, Jun 3, 2019 at 6:19 PM wrote: > Kevin, > > > > I'm using Livy 0.6.0. The issues is related to not finding repl jars that > support scala 2.12. The error "requirement failed: Cannot find Livy REPL > jars." is thrown because it couldn't find folder repl_2.12-jars under > LIVY directory. > > > > I performed a test to make sure this issue is related to scala 2.12 > compatibility , I copied contents of repl_2.11-jars under Livy directory > into new directory LIVY/repl_2.12-jars and this time I didn't get REPL jars > exception it went ahead and created session but failed to start session due > to rsc jars version incompatibility. > > > > *LIVY Folder structure for error " requirement failed: Cannot find Livy > REPL jars.""* > > > > [/app/risk/ha02/livy]$ ls -ltr > > total 116 > > -rwxr-xr-x 1 agriddev agriddev 160 Mar 19 14:39 NOTICE > > -rwxr-xr-x 1 agriddev agriddev 18665 Mar 19 14:39 LICENSE > > -rwxr-xr-x 1 agriddev agriddev 537 Mar 19 14:39 DISCLAIMER > > -rwxr-xr-x 1 agriddev agriddev 46355 Mar 19 14:42 THIRD-PARTY > > drwxr-xr-x 2 agriddev agriddev 4096 Mar 19 14:43 bin > > drwxr-xr-x 2 agriddev agriddev 4096 Apr 14 22:37 repl_2.11-jars > > drwxr-xr-x 2 agriddev agriddev 4096 Apr 14 22:37 rsc-jars > > drwxr-xr-x 2 agriddev agriddev 12288 Apr 14 22:37 jars > > drwxr-xr-x 2 agriddev agriddev 4096 Apr 14 22:37 > apache-livy-0.6.0-incubating-bin > > drwxr-xr-x 2 agriddev agriddev 4096 Jun 3 17:37 conf > > drwxr-xr-x 2 agriddev agriddev 4096 Jun 3 21:51 logs > > > > *LIVY FOLDER STRUCTURE TO BYPASS "REQUIREMENT FAILED:CANNOT FIND LIVY REPL > JARS"* > > > > [/app/risk/ha02/livy]$ ls -ltr > > total 116 > > -rwxr-xr-x 1 agriddev agriddev 160 Mar 19 14:39 NOTICE > > -rwxr-xr-x 1 agriddev agriddev 18665 Mar 19 14:39 LICENSE > > -rwxr-xr-x 1 agriddev agriddev 537 Mar 19 14:39 DISCLAIMER > > -rwxr-xr-x 1 agriddev agriddev 46355 Mar 19 14:42 THIRD-PARTY > > drwxr-xr-x 2 agriddev agriddev 4096 Mar 19 14:43 bin > > drwxr-xr-x 2 agriddev agriddev 4096 Apr 14 22:37 repl_2.11-jars > > drwxr-xr-x 2 agriddev agriddev 4096 Apr 14 22:37 rsc-jars > > drwxr-xr-x 2 agriddev agriddev 12288 Apr 14 22:37 jars > > drwxr-xr-x 2 agriddev agriddev 4096 Apr 14 22:37 > apache-livy-0.6.0-incubating-bin > > drwxr-xr-x 2 agriddev agriddev 4096 Jun 3 17:37 conf > > drwxr-xr-x 2 agriddev agriddev 4096 Jun 3 21:50 repl_2.12-jars > > drwxr-xr-x 2 agriddev agriddev 4096 Jun 3 21:51 logs > > > > > > Error Information > > > > zip > > 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO > SecurityManager: Changing view acls to: agriddev > > 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO > SecurityManager: Changing modify acls to: agriddev > > 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO > SecurityManager: Changing view acls groups to: > > 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO > SecurityManager: Changing modify acls groups to: > > 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO > SecurityManager: SecurityManager: authentication disabled; ui acls > disabled; users with view permissions: Set(agriddev); groups with view > permissions: Set(); users with modify permissions: Set(agriddev); groups > with modify permissions: Set() > > 19/06/03 21:52:01 INFO LineBufferedStream: 19/06/03 21:52:01 INFO Client: > Submitting application application_1559316432251_0172 to ResourceManager > > 19/06/03 21:52:01 INFO LineBufferedStream: 19/06/03 21:52:01 INFO > YarnClientImpl: Submitted application application_1559316432251_0172 > > 19/06/03 21:52:01 INFO LineBufferedStream: 19/06/03 21:52:01 INFO Client: > Application report for application_1559316432251_0172 (state: ACCEPTED) > > 19/06/03 21:52:01 INFO LineBufferedStream: 19/06/03 21:52:01 INFO Client: > > 19/06/03 21:52:01 INFO LineBufferedStream: client token: N/A > > 19/06/03 21:52:01 INFO LineBufferedStream: diagnostics: [Mon Jun 03 > 21:52:01 + 2019] Application is Activated, waiting for resources to be > assigned for AM. Details : AM Partition = ; Partition > Resource = ; Queue's Absolute capacity = 100.0 > % ; Queue's Absolute used capacity = 0.0 % ; Queue's Absolute max capacity > = 100.0 % ; > > 19/06/03 21:52:01 INFO LineBufferedStream:
Re: Support for Livy with Scala 2.12
"requirement failed: Cannot find Livy REPL jars."
I didn't look where that error comes from, but my guess is that it looks
like you don't have Livy pointing to the right location anymore where it
can find Spark. Hopefully not sending you on a wild goose chase, but would
check there first.
Also you need to make sure you are Livy 0.6.0+ since Spark 2.4 support was
only added to 0.6.0 and above.
Kevin Risden
On Mon, Jun 3, 2019 at 4:04 PM Pat Ferrel wrote:
> Spark 2.4.x does not require scala 2.12, in fact is is marked as
> “experimental” here:
> https://spark.apache.org/releases/spark-release-2-4-0.html
>
> <https://spark.apache.org/releases/spark-release-2-4-0.html>
> Moving to a new scala version is often a pain, because the libs you use
> may not be upgraded and version matter *unlike typical Java updates). Scala
> creates JVM objects and names them as it pleases. Sometimes naming changes
> from version to version of Scala and this causes big problem in using mixed
> libs from different versions of Scala.
>
> I’m no expert in Livy, but imagine you may need to build against a newer
> Spark. But avoid Scala 2.12 for now.
>
> From: santosh.dan...@ubs.com
>
> Reply: user@livy.incubator.apache.org
>
> Date: June 3, 2019 at 12:51:20 PM
> To: user@livy.incubator.apache.org
>
> Subject: Support for Livy with Scala 2.12
>
> Hi,
>
>
>
> We have just upgraded our spark cluster version 2.3 to 2.4.2 and it broke
> Livy. It's throwing exception "Cannot Find Livy REPL Jars". Looks like I
> have to build Livy using Scala 2.12 version.
>
>
>
> Can anyone advise how to build Livy with Scala 2.12 with Maven? Will
> changing the scala version from 2.11 to 2.12 would build livy? Please
> advise.
>
>
>
>
>
>
>
> The code failed because of a fatal error:
>
> Invalid status code '400' from http://localhost:8998/sessions with
> error payload: {"msg":"requirement failed: Cannot find Livy REPL jars."}.
>
>
>
> Thanks
> Santosh
>
> Please visit our website at
> http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html
> for important disclosures and information about our e-mail
> policies. For your protection, please do not transmit orders
> or instructions by e-mail or include account numbers, Social
> Security numbers, credit card numbers, passwords, or other
> personal information.
>
>
Re: appending @realm to usernames
I don't think I follow your statement that @realm is mandatory. Auth to local is basically just a regex. https://community.hortonworks.com/articles/14463/auth-to-local-rules-syntax.html I don't know why you want to append the realm back anyway since usually the username is what you are after anyway. Kevin Risden On Tue, Jan 15, 2019 at 12:36 PM Ruslan Dautkhanov wrote: > > We'd like Hadoop to map user names to short names. > > For auth_to_local to work, @realm part is mandatory. > > For example, Apache Knox if authenticates users using LDAP, > and then sends requests over to Livy, doesn't append realm. > Obviously LDAP, PAM etc authentications don't have kerberos > realms there. > > Is there is a way for append realm in Livy, before it sends > those requests over to Spark / Hadoop? > > It seems we could duplicate rules from Hadoop's auth_to_local > using `livy.server.auth.kerberos.name_rules` but it doesn't work > for the same reason (kerberos rules requires realm to be present). > > Also created https://issues.apache.org/jira/browse/LIVY-548 > > Thank you for any ideas. > > -- > Ruslan Dautkhanov
Re: Enable HTTP Basic Authenticaton for Livy Rest APIs
One option that is available today is to use Apache Knox [1]. Apache Knox supports Livy [2] as a backend and a lot of different options for authentication [3]. 1. https://knox.apache.org/ 2. https://knox.apache.org/books/knox-1-1-0/user-guide.html#Livy+Server 3. https://knox.apache.org/books/knox-1-1-0/user-guide.html#Authentication Kevin Risden On Fri, Oct 5, 2018 at 7:05 PM Santosh Dandey wrote: > Thanks > > On Fri, Oct 5, 2018, 6:03 PM Marcelo Vanzin wrote: > >> As I mentioned in my e-mail, there's already work to support generic auth >> methods. Once that's in you can plug in your own authentication filter to >> Livy. >> >> On Fri, Oct 5, 2018 at 4:00 PM Santosh Dandey >> wrote: >> >>> Vanzin, >>> >>> I would like to implement this change to tailor our Enterprise >>> requirements. >>> >>> Is there any developments guidelined available to get me started to >>> change the code? >>> >>> On Fri, Oct 5, 2018, 2:50 PM Marcelo Vanzin wrote: >>> >>>> I don't think Livy currently has support for basic auth. >>>> >>>> Although this PR would allow you to set up your own custom auth >>>> configuration: >>>> https://github.com/apache/incubator-livy/pull/110 >>>> >>>> >>>> On Thu, Oct 4, 2018 at 2:34 PM wrote: >>>> >>>>> I'm Santosh Dandey, I'm working at UBS as one of the lead-developer >>>>> for a Big Data Project. We are working on enabling Jupyter notebook as a >>>>> data-science tool for data-scientists. I have below setup that's working >>>>> good in my local environment without any authentication between >>>>> spark-magic >>>>> and Livy. >>>>> >>>>> I have to implement security for SparkMagic and Apache Livy. On Spark >>>>> Magic it says I can enable HTTP Basic Authentication if it's enabled in >>>>> Livy. How do I enable HTTP Basic authentication in livy? >>>>> >>>>> >>>>> >>>>> How can we enable basic-authentication on Livy? Is there any specific >>>>> configuration that I have to set on livy.conf to enable basic >>>>> authentication. >>>>> >>>>> >>>>> >>>>> Did anyone had luck implementing basic authentication? I appreciate >>>>> if you can provide any help here. >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Santosh >>>>> >>>>> >>>>> >>>>> [image: cid:image003.png@01D45BF9.4FC434F0] >>>>> >>>> >>>> >>>> -- >>>> Marcelo >>>> >>> >> >> -- >> Marcelo >> >
