Thanks Josef and Victor. It is working. Thanks!
2015-11-04 12:36 GMT+01:00 Josef A. Habdank <jahabd...@gmail.com>: > It worked for me. > > I have put a small help, which I might post to some blog at some point > (how to setup S3 access for Zeppelin on EMR): > > To have default EMR instance with Zeppelin store notebooks in S3 > > # in S3 create the following bucket: > > /my-bucket-name/zeppelin/notebook/ > > # edit the /etc/zeppelin/conf.dist/zeppelin-env.sh > > sudo vim /etc/zeppelin/conf.dist/zeppelin-env.sh > > # add: > > export ZEPPELIN_NOTEBOOK_S3_BUCKET=my-bucket-name > > export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin > > # create and edit the zeppelin-site.xml > > sudo cp /etc/zeppelin/conf.dist/zeppelin-site.xml.template > /etc/zeppelin/conf.dist/zeppelin-site.xml > > sudo vim /etc/zeppelin/conf.dist/zeppelin-site.xml > > # uncomment: > > <property> > > <name>zeppelin.notebook.storage</name> > > <value>org.apache.zeppelin.notebook.repo.S3NotebookRepo</value> > > <description>notebook persistence layer implementation</description> > > </property> > > > # comment: > > <property> > > <name>zeppelin.notebook.storage</name> > > <value>org.apache.zeppelin.notebook.repo.VFSNotebookRepo</value> > > <description>notebook persistence layer implementation</description> > > </property> > > # remember to put the notebooks in the S3 :) > > # can download from here: https://github.com/Leemoonsoo?tab=repositories > > # restart zeppelin > > sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh stop > > sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh start > > # should work :) > > On 4 November 2015 at 12:21, Pablo Torre <pablotorr...@gmail.com> wrote: > >> Thanks Victor, I appreciate your help, I will give a try. >> >> >> >> >> 2015-11-04 12:11 GMT+01:00 Victor Manuel Garcia <victor.gar...@beeva.com> >> : >> >>> yes, the username is the same in zeppelin-site.xml and the bucket you >>> need specify only the name. >>> >>> 2015-11-04 12:03 GMT+01:00 Pablo Torre <pablotorr...@gmail.com>: >>> >>>> Ok, because I am trying to test it from my localhost using a bucket in >>>> Amazon S3. So if I am right I need to set up those two environment >>>> variables in my localhost: >>>> >>>> ENV AWS_ACCESS_KEY=key >>>> ENV AWS_SECRET_KEY=secret >>>> >>>> The other question that I have is about those two variables: >>>> >>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=bucket >>>> export ZEPPELIN_NOTEBOOK_S3_USER=username >>>> >>>> The username I guess is the same user that you specify in >>>> zeppelin-site.xml, and the bucket is only the name, I don't need to specify >>>> s3://bucket.... >>>> >>>> Please let me know if I am wrong >>>> >>>> Best, >>>> Pablo >>>> >>>> >>>> 2015-11-04 11:59 GMT+01:00 Victor Manuel Garcia < >>>> victor.gar...@beeva.com>: >>>> >>>>> Hi Pablo, >>>>> Yes, you need the AWS environment variables o credentials file in .aws >>>>> folder in your home... >>>>> >>>>> 2015-11-04 11:34 GMT+01:00 Pablo Torre <pablotorr...@gmail.com>: >>>>> >>>>>> Hi Victor, >>>>>> >>>>>> Taking a look at your suggestion, I was wondering if we need to >>>>>> export the AWS Credentials..... >>>>>> >>>>>> Best, >>>>>> Pablo >>>>>> >>>>>> 2015-11-03 10:46 GMT+01:00 Victor Manuel Garcia < >>>>>> victor.gar...@beeva.com>: >>>>>> >>>>>>> Hi Josef, >>>>>>> For use S3 you need: >>>>>>> >>>>>>> Folder structure needs S3: >>>>>>> bucket/username/notebook/ >>>>>>> >>>>>>> set the enviroment variable: >>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=bucket >>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=username >>>>>>> >>>>>>> and in zeppelin-site.xml: >>>>>>> <!-- If used S3 to storage the notebooks, it is necessary the >>>>>>> following folder structure bucketname/username/notebook/ --> >>>>>>> <!-- >>>>>>> <property> >>>>>>> <name>zeppelin.notebook.s3.user</name> >>>>>>> <value>user</value> >>>>>>> <description>user name for s3 folder structure</description> >>>>>>> </property> >>>>>>> <property> >>>>>>> <name>zeppelin.notebook.s3.bucket</name> >>>>>>> <value>zeppelin</value> >>>>>>> <description>bucket name for notebook storage</description> >>>>>>> </property> >>>>>>> >>>>>>> Uncomment: >>>>>>> <property> <name>zeppelin.notebook.storage</name> >>>>>>> <value>org.apache.zeppelin.notebook.repo.S3NotebookRepo</value> >>>>>>> <description>notebook persistence layer implementation</description> >>>>>>> </property> >>>>>>> >>>>>>> Comment: >>>>>>> <property> >>>>>>> <name>zeppelin.notebook.storage</name> >>>>>>> <value>org.apache.zeppelin.notebook.repo.VFSNotebookRepo</value> >>>>>>> <description>notebook persistence layer >>>>>>> implementation</description> >>>>>>> </property> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2015-11-03 10:11 GMT+01:00 Josef A. Habdank <jahabd...@gmail.com>: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> on the presentation on Spark Summit Amsterdam Moon showcased that >>>>>>>> it is possible to change a location of the storage of the notebooks to >>>>>>>> S3 >>>>>>>> (so they are not stored locally on the zeppelin machine). >>>>>>>> >>>>>>>> I am running a standard Amazon EMR cluster with Zeppelin 0.6 >>>>>>>> SNAPSHOT (default Amazon installation). >>>>>>>> >>>>>>>> I tried to modify the /etc/zeppelin/conf.dist/zeppelin-env.sh file >>>>>>>> changing >>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook >>>>>>>> >>>>>>>> to the following: >>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=s3:/bucket/folder >>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=s3://bucket/folder >>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=s3n://bucket/folder >>>>>>>> >>>>>>>> and each time when I restart Zeppelin using >>>>>>>> sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh stop >>>>>>>> sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh start >>>>>>>> >>>>>>>> I get error: >>>>>>>> "Notebook dir doesn't exist, create" >>>>>>>> >>>>>>>> when looking in to the zeppelin-env.sh I noticed it only checks for >>>>>>>> the local folder. >>>>>>>> if [[ ! -d "${ZEPPELIN_NOTEBOOK_DIR}" ]]; then >>>>>>>> echo "Notebook dir doesn't exist, create >>>>>>>> ${ZEPPELIN_NOTEBOOK_DIR}" >>>>>>>> $(mkdir -p "${ZEPPELIN_NOTEBOOK_DIR}") >>>>>>>> fi >>>>>>>> >>>>>>>> So I assume it is not the way :) >>>>>>>> >>>>>>>> Can you please tell me how can I do it? I want to store all my >>>>>>>> notebooks on s3 folder, rather than locally. >>>>>>>> >>>>>>>> Thanks a lot! >>>>>>>> >>>>>>>> Josef >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Victor Manuel Garcia Martinez* >>>>>>> *Ingeniero de Software >>>>>>> * >>>>>>> >>>>>>> *+34 672104297 <%2B34%20672104297> | victor.gar...@beeva.com >>>>>>> <marta.ta...@beeva.com>* >>>>>>> * | victormanuel.garcia.marti...@bbva.com >>>>>>> <marta.ta...@bbva.com>* >>>>>>> >>>>>>> >>>>>>> >>>>>>> <http://www.beeva.com/> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pablo Torre. >>>>>> Freelance software engineer and Ruby on Rails developer. >>>>>> Oleiros (Coruña) >>>>>> *Personal site <http://www.pablotorrerodriguez.com>* >>>>>> My blog <http://www.aboutechnologies.com> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Victor Manuel Garcia Martinez* >>>>> *Ingeniero de Software >>>>> * >>>>> >>>>> *+34 672104297 <%2B34%20672104297> | victor.gar...@beeva.com >>>>> <marta.ta...@beeva.com>* >>>>> * | victormanuel.garcia.marti...@bbva.com >>>>> <marta.ta...@bbva.com>* >>>>> >>>>> >>>>> >>>>> <http://www.beeva.com/> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Pablo Torre. >>>> Freelance software engineer and Ruby on Rails developer. >>>> Oleiros (Coruña) >>>> *Personal site <http://www.pablotorrerodriguez.com>* >>>> My blog <http://www.aboutechnologies.com> >>>> >>> >>> >>> >>> -- >>> *Victor Manuel Garcia Martinez* >>> *Ingeniero de Software >>> * >>> >>> *+34 672104297 <%2B34%20672104297> | victor.gar...@beeva.com >>> <marta.ta...@beeva.com>* >>> * | victormanuel.garcia.marti...@bbva.com >>> <marta.ta...@bbva.com>* >>> >>> >>> >>> <http://www.beeva.com/> >>> >>> >>> >> >> >> -- >> Pablo Torre. >> Freelance software engineer and Ruby on Rails developer. >> Oleiros (Coruña) >> *Personal site <http://www.pablotorrerodriguez.com>* >> My blog <http://www.aboutechnologies.com> >> > > -- Pablo Torre. Freelance software engineer and Ruby on Rails developer. Oleiros (Coruña) *Personal site <http://www.pablotorrerodriguez.com>* My blog <http://www.aboutechnologies.com>
