%spark z.load() didn't have such limitation. So if we can implement it correctly, it would be really handy.
Thanks, moon On Sat, Feb 20, 2016 at 8:58 AM Felix Cheung <felixcheun...@hotmail.com> wrote: > Would %spark z.load() have the earlier limitation of that it must be run > before anything to do with spark? > > > > > > On Sat, Feb 20, 2016 at 12:33 AM -0800, "Ankur Jain" <ankur.j...@yash.com> > wrote: > > LGTM… > > > > Thanks > > Ankur > > > > *From:* moon soo Lee [mailto:m...@apache.org] > *Sent:* 20 February 2016 11:24 AM > *To:* users@zeppelin.incubator.apache.org > *Subject:* Re: Unable to access AVRO method > > > > UI way requires interpreter process restart. So the behavior will be > different from %spark z.load() you use right now. And...new UI way keeps > list of dependencies. It does not reset after interpreter/zeppelin restart. > So without api for removing dependency, dependencies will never decrease. > > > > Hence, to me, the best option seems > > > > - bringing %spark z.load() back with correct implementation > > - deprecate and remove %dep z.load() while it was alternative for buggy > %spark z.load() > > - keep new UI way, while not only Spark but all other interpreters can > leverage it. > > > > > > Thanks, > > moon > > > > On Fri, Feb 19, 2016 at 2:30 PM Aish Fenton <afen...@netflix.com> wrote: > > > > I mean add in the dependency in the same way that the UI does. > > > > > > On Fri, Feb 19, 2016 at 2:30 PM Aish Fenton <afen...@netflix.com> wrote: > > > > Hmm, > > Maybe you could change the semantics of z.load to be automatically add in > the dependency? The only advantage I was really after with z.load was that > self-documenting feature. > > > > > > > > > > On Fri, Feb 19, 2016 at 2:11 PM moon soo Lee <m...@apache.org> wrote: > > Right, new UI provides more generalized way for dependency loading for all > interpreters. not only spark but also jdbc, etc. > > > > For spark, > > > > We have had %spark z.load() at first, but it was too buggy, so we have > created %dep z.load(), which is far less buggy but still some library > couldn't loaded in this way. Finally, the new UI way, handles all the case > correctly. > > > > I agree if we could have z.load for spark, it's self-documenting. And it > may be convenient for multiple people working. > > > > If you're interested, i can work on bringing %spark z.load() back. But > then we'll have 3 different way of dependency loading for Spark interpreter. > > > > - %spark z.load() (removed in master branch) > > - %dep z.load() (marked as deprecated in master branch) > > - new UI way. > > > > I think new UI way is for all interpreters. so we definitely want to keep > it. > > So i want to ask people which one do we want to keep? %spark z.load(), > %dep z.load(). Neither, both, one of it? > > > > Thanks, > > moon > > > > On Fri, Feb 19, 2016 at 10:52 AM Aish Fenton <afen...@netflix.com> wrote: > > > > z.load was buggy, but it was self-documenting. I'm wondering with the new > UI way, how will this work for people sharing/importing notebook? We quite > commonly share notebooks. > > > > > > > > On Fri, Feb 19, 2016 at 10:09 AM moon soo Lee <m...@apache.org> wrote: > > Hi Ankur, > > > > It's bit depends on you Zeppelin version, but basically > > z.load() inside of Spark interpreter was buggy and we have now removed > from master branch. > > > > You can check "Dependency Management" section, from > http://zeppelin.incubator.apache.org/docs/latest/interpreter/spark.html > > > > Basically, two different way > > one using %dep interpreter, the other using spark properties. > > > > If you can use master branch, you can load dependencies from Interpreter > menu via GUI. > > > > Hope this helps. > > > > Best, > > moon > > On Fri, Feb 19, 2016 at 5:50 AM Ankur Jain <ankur.j...@yash.com> wrote: > > Hi Team, > > > > I loaded avro dependencies, and then tried to read avro files, but it is > throwing below exception… > > > > [image: image001.jpg] > > > > Can you please guide on this? > > > > My Environment details are as mentioned below… > > > https://blogs.aws.amazon.com/bigdata/post/Tx2HJD3Z74J2U8U/Running-an-External-Zeppelin-Instance-using-S3-Backed-Notebooks-with-Spark-on-Am > > > > Thanks, > > Ankur > > Information transmitted by this e-mail is proprietary to YASH Technologies > and/ or its Customers and is intended for use only by the individual or > entity to which it is addressed, and may contain information that is > privileged, confidential or exempt from disclosure under applicable law. If > you are not the intended recipient or it appears that this mail has been > forwarded to you without proper authority, you are notified that any use or > dissemination of this information in any manner is strictly prohibited. In > such cases, please notify us immediately at i...@yash.com and delete this > mail from your records. > > Information transmitted by this e-mail is proprietary to YASH Technologies > and/ or its Customers and is intended for use only by the individual or > entity to which it is addressed, and may contain information that is > privileged, confidential or exempt from disclosure under applicable law. If > you are not the intended recipient or it appears that this mail has been > forwarded to you without proper authority, you are notified that any use or > dissemination of this information in any manner is strictly prohibited. In > such cases, please notify us immediately at i...@yash.com and delete this > mail from your records. >
