On Mon, Aug 1, 2016 at 10:16 AM, 薛亚兰 <[email protected]> wrote:
> Thanks MinRK, found the problem of this notebook.dir issue.
> And another problem came when I used the old way as 00-pyspark-setup.py to
> start pyspark with Jupyter notebook. We want to combine Spark 1.6.1 and
> Jupyter notebook for usage, and originally, we use ipython 3.2.1, and use
> python setup.py to install it, then use ipython profile create to create
> all the configure files and also it will produce the startup dir in the
> profile_dir, then we put the 00-pyspark-setup.py in startup dir, then the
> sparkcontext will be produced successfully.
>
> However, currently, we want to move to jupyter 4.1, and use Anaconda to
> help us to install it, but seems the 00-pyspark-setup.py file does not
> executed anymore, the following are my configurations:
>
> export PYSPARK_SUBMIT_ARGS="--deploy-mode client --master $masterurl
> pyspark-shell"
> PYSPARK_PYTHON=Anaconda/installdir/bin/python
> PYSPARK_DRIVER_PYTHON=Anaconda/installdir/bin/jupyter
>
> and then create dir $IPYTHONDIR/profile_default/startup/, and then put the
> 00-pyspark-setup.py
> in it, using cmds like:
> cp 00-pyspark-setup.py $IPYTHONDIR/profile_default/startup/
>
> and then start the jupyter notebook:
> jupyter notebook --notebook-dir=$JUPYTER_NOTEBOOK_DIR
>
> I just see all the files are produced as structures like this in
> IPYTHONDIR:
>
> drwx------. 3 root root 67 Aug 1 15:38 config
> drwx------. 2 root root 6 Aug 1 15:40 extensions
> -rw-------. 1 root root 2415 Aug 1 15:42 ipythonlog
> drwx------. 2 root root 6 Aug 1 15:40 nbextensions
> drwx------. 3 root root 52 Aug 1 15:42 notebooks
> drwx------. 7 root root 85 Aug 1 15:58 profile_default
>
> *in profile_default dir:*
> drwx------. 2 root root 6 Aug 1 15:40 db
> -rw-------. 1 root root 28672 Aug 1 15:40 history.sqlite
> drwx------. 2 root root 6 Aug 1 15:40 log
> drwx------. 2 root root 6 Aug 1 15:40 pid
> drwx------. 2 root root 6 Aug 1 15:40 security
> drwx------. 2 root root 45 Aug 1 15:40 startup
>
> but however, notebook APP can be started well, and we can access its urls,
> but unfortunately, the sparkcontext is not initialized successfully, I can
> guarantee the master url and spark_home is set right and both of them works
> well. But did something changed in Jupyter compared to original ipython, or
> did I miss some steps?
>
IPython profiles have not changed with the transition to Jupyter. The only
difference is that Jupyter itself (the notebook server) is no longer
configured by the IPython config, only the IPython kernel is.
>
> Really need support from you, I am blocked by this.
> the following are the content of my 00-pyspark-setup.py file:
>
> import os
> import sys
>
> spark_home = os.environ.get('SPARK_HOME', None)
> if not spark_home:
> raise ValueError('SPARK_HOME environment variable is not set')
> sys.path.insert(0, os.path.join(spark_home, 'python'))
> tmpDir = os.path.join(spark_home,'python/lib/')
> tmpStr = "py4j"
> joinDir = tmpDir
> for root, dirs, files in os.walk(tmpDir):
> for f in files:
> if tmpStr in f:
> joinDir = os.path.join(tmpDir,f)
> break
> if joinDir == tmpDir:
> raise ValueError('py4j library is not set in spark_home dir')
>
> sys.path.insert(0, joinDir)
> execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))
>
> Thanks in advance!
>
Can you verify that any of this file is executed, e.g. is `spark_home`
defined? Is `joinDir` defined? Can you start a regular IPython terminal int
he same context, so that it will load this same profile? One disadvantage
of the notebook is that errors raised in your startup files will not be
easily visible.
-MinRK
>
>
>
> 2016-07-28 17:00 GMT+08:00 MinRK <[email protected]>:
>
>> On Thu, Jul 28, 2016 at 9:32 AM, 薛亚兰 <[email protected]> wrote:
>>
>> Thanks Thomas, then I knew how to put the custom.js file, and I tried, it
>>> worked, Thanks again.
>>>
>>> however, I have another question about the notebook local directory,
>>> what parameter we should set to change it, since in my example, I really
>>> have no idea why the local directory of mine should be this value,
>>> Serving notebooks from local directory: */tmp/nonroot/jupyter/work*
>>> I run the jupyter notebook at this dir */tmp/nonroot/jupyter/,* and I
>>> already set the *c.NotebookApp.notebook_dir* to be another value, but
>>> the local directory keeps the same, everytime I start a notebook, it is
>>> this value
>>>
>>> */tmp/nonroot/jupyter/work/. *
>>> Do you have any idea about these above dir associated issues?
>>>
>> Setting c.NotebookApp.notebook_dir should indeed set the default
>> notebook directory. This will be overridden by command-line arguments, e.g.
>>
>> jupyter notebook /path/to/notebooks
>>
>> or
>>
>> jupyter notebook --notebook-dir /path/to/notebooks
>>
>> You might verify that this config is picked up, by starting the notebook
>> server with --debug.
>>
>> -MinRK
>>
>>
>>> Thanks in advance!
>>>
>>> 2016-07-28 4:40 GMT+08:00 Thomas Kluyver <[email protected]>:
>>>
>>>> On 27 July 2016 at 11:36, <[email protected]> wrote:
>>>>
>>>>> the config file really produced in MY_CONFIG_DIR, and some files like
>>>>> nbserver-20374.json produced in runtime dir of MY_DATA_DIR, but I
>>>>> definitely can not find the dirs as custom and files as custom.js, Could
>>>>> anyone help me or tell me where are the files' location?
>>>>>
>>>>
>>>> The custom files are not created automatically, but you can put them at
>>>> e.g. $JUPYTER_CONFIG_DIR/custom/custom.js
>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "Project Jupyter" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/jupyter/VF5RHpncN2k/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> [email protected].
>>>> To post to this group, send email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/jupyter/CAOvn4qh%2BkSQ_cdGj6M28ZanoXJTNpEgiTrT7AWYc%3D-5XX_anww%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/jupyter/CAOvn4qh%2BkSQ_cdGj6M28ZanoXJTNpEgiTrT7AWYc%3D-5XX_anww%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> Yalan Xue ， SIT Lab of Nanjing University
>>> Address: Nanjing ,China。
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Project Jupyter" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/jupyter/CACNymyti%3DfBzxrU56qTk3fMaeFPYit%3DcwQUoYMve2_%2B9VqqCyg%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/jupyter/CACNymyti%3DfBzxrU56qTk3fMaeFPYit%3DcwQUoYMve2_%2B9VqqCyg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Project Jupyter" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/jupyter/VF5RHpncN2k/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected].
>> To post to this group, send email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/CAHNn8BUJ%2BUz-UYqwb4se0Z_Vux08f-d8xLLKRoVw_q9d8DQpZg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/jupyter/CAHNn8BUJ%2BUz-UYqwb4se0Z_Vux08f-d8xLLKRoVw_q9d8DQpZg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Yalan Xue ， SIT Lab of Nanjing University
> Address: Nanjing ,China。
>
> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jupyter/CACNymysQj1%3DaDTbeoedmYaqsX_sKMdxAKYb1Pbq9MzeUx-Mdig%40mail.gmail.com
> <https://groups.google.com/d/msgid/jupyter/CACNymysQj1%3DaDTbeoedmYaqsX_sKMdxAKYb1Pbq9MzeUx-Mdig%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/CAHNn8BUwx_5CNJBiDNgXUk_e12mL0o5mSwZ2m%3D1naLBQ9OSXGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
