I ended up having a notebook called “shared_code”, and would just always run it before I did any work. That approach worked, but was error prone and made refactoring any complex logic much harder. I did play around a bit with some of the %sh to edit .py files, and then load them..
Jeff, are there any examples of what you meant by using the python interpreter? > On Jan 10, 2020, at 6:31 PM, Jeff Zhang <zjf...@gmail.com > <mailto:zjf...@gmail.com>> wrote: > > You can do it via ipython interpreter which support all the of jupyter magics > > http://zeppelin.apache.org/docs/0.8.2/interpreter/python.html#ipython-support > <http://zeppelin.apache.org/docs/0.8.2/interpreter/python.html#ipython-support> > > > Partridge, Lucas (GE Aviation) <lucas.partri...@ge.com > <mailto:lucas.partri...@ge.com>> 于2020年1月10日周五 下午5:13写道： > I've hardly used Jupyter so can't comment on an equivalent for %run. > > But for Zeppelin you can put your python files on the local file system of > your Spark driver node, or more commonly in HDFS, and then use sc.addPyFile() > [1] to make each file available in the SparkContext. Then you can import > your python packages as normal. The slightly annoying thing is that if you > change your code you'll need to restart your Spark application to pick up the > changes as there's no reliable way to reimport the updated modules in a > running application. But you could put your importing of common files in a > shared notebook so everyone can run it easily. > > Once you're happy with your code and it's fairly stable then you can package > it with a setup.py and install the packages on all the nodes of your cluster > like any other python package. Then you can skip the sc.addPyFile() step. > > DataBricks have a great facility for allowing users to upload their own > Python packages/libraries. It would be great if Zeppelin provided this > feature as well (although maybe they do now as I'm on an older version...). > > Lucas. > > [1] > https://spark.apache.org/docs/latest/api/python/pyspark.html?highlight=addpyfile#pyspark.SparkContext.addPyFile > > <https://spark.apache.org/docs/latest/api/python/pyspark.html?highlight=addpyfile#pyspark.SparkContext.addPyFile> > > -----Original Message----- > From: Dave Boyd <db...@incadencecorp.com <mailto:db...@incadencecorp.com>> > Sent: 09 January 2020 17:44 > To: users@zeppelin.apache.org <mailto:users@zeppelin.apache.org> > Subject: EXT: Equivalent of Jupyter %run > > I have googled this but don't see a solution. > > We are working on a project where we want to have some common python > functions shared between notes. > > In Jupyter we would just do a %run. Is there an equivelent in Zeppelin? > Is there a way to store files as .py files that zeppelin can find for import > to work. > > Looking to see how folks may have solved this need. > > -- > ========= mailto:db...@incadencecorp.com <mailto:db...@incadencecorp.com> > ============ David W. Boyd VP, Data Solutions > 10432 Balls Ford, Suite 240 > Manassas, VA 20109 > office: +1-703-552-2862 > cell: +1-703-402-7908 > ============== http://www.incadencecorp.com/ <http://www.incadencecorp.com/> > ============ ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 > Chair ANSI/INCITS TG Big Data Co-chair NIST Big Data Public Working Group > Reference Architecture First Robotic Mentor - FRC, FTC - > www.iliterobotics.org <http://www.iliterobotics.org/> Board Member- USSTEM > Foundation - www.usstem.org <http://www.usstem.org/> > > The information contained in this message may be privileged and/or > confidential and protected from disclosure. > If the reader of this message is not the intended recipient or an employee or > agent responsible for delivering this message to the intended recipient, you > are hereby notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you have received this > communication in error, please notify the sender immediately by replying to > this message and deleting the material from any computer. > > > > -- > Best Regards > > Jeff Zhang _______________________ Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
