Thanks MinRK, found the problem of this notebook.dir issue.
And another problem came when I used the old way as 00-pyspark-setup.py to
start pyspark with Jupyter notebook. We want to combine Spark 1.6.1 and
Jupyter notebook for usage, and originally, we use ipython 3.2.1, and use
python setup.py to install it, then use ipython profile create to create
all the configure files and also it will produce the startup dir in the
profile_dir, then we put the 00-pyspark-setup.py in startup dir, then the
sparkcontext will be produced successfully.
However, currently, we want to move to jupyter 4.1, and use Anaconda to
help us to install it, but seems the 00-pyspark-setup.py file does not
executed anymore, the following are my configurations:
export PYSPARK_SUBMIT_ARGS="--deploy-mode client --master $masterurl
pyspark-shell"
PYSPARK_PYTHON=Anaconda/installdir/bin/python
PYSPARK_DRIVER_PYTHON=Anaconda/installdir/bin/jupyter
and then create dir $IPYTHONDIR/profile_default/startup/, and then put
the 00-pyspark-setup.py
in it, using cmds like:
cp 00-pyspark-setup.py $IPYTHONDIR/profile_default/startup/
and then start the jupyter notebook:
jupyter notebook --notebook-dir=$JUPYTER_NOTEBOOK_DIR
I just see all the files are produced as structures like this in
IPYTHONDIR:
drwx------. 3 root root 67 Aug 1 15:38 config
drwx------. 2 root root 6 Aug 1 15:40 extensions
-rw-------. 1 root root 2415 Aug 1 15:42 ipythonlog
drwx------. 2 root root 6 Aug 1 15:40 nbextensions
drwx------. 3 root root 52 Aug 1 15:42 notebooks
drwx------. 7 root root 85 Aug 1 15:58 profile_default
*in profile_default dir:*
drwx------. 2 root root 6 Aug 1 15:40 db
-rw-------. 1 root root 28672 Aug 1 15:40 history.sqlite
drwx------. 2 root root 6 Aug 1 15:40 log
drwx------. 2 root root 6 Aug 1 15:40 pid
drwx------. 2 root root 6 Aug 1 15:40 security
drwx------. 2 root root 45 Aug 1 15:40 startup
but however, notebook APP can be started well, and we can access its urls,
but unfortunately, the sparkcontext is not initialized successfully, I can
guarantee the master url and spark_home is set right and both of them works
well. But did something changed in Jupyter compared to original ipython, or
did I miss some steps?
Really need support from you, I am blocked by this.
the following are the content of my 00-pyspark-setup.py file:
import os
import sys
spark_home = os.environ.get('SPARK_HOME', None)
if not spark_home:
raise ValueError('SPARK_HOME environment variable is not set')
sys.path.insert(0, os.path.join(spark_home, 'python'))
tmpDir = os.path.join(spark_home,'python/lib/')
tmpStr = "py4j"
joinDir = tmpDir
for root, dirs, files in os.walk(tmpDir):
for f in files:
if tmpStr in f:
joinDir = os.path.join(tmpDir,f)
break
if joinDir == tmpDir:
raise ValueError('py4j library is not set in spark_home dir')
sys.path.insert(0, joinDir)
execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))
Thanks in advance!
2016-07-28 17:00 GMT+08:00 MinRK <[email protected]>:
> On Thu, Jul 28, 2016 at 9:32 AM, 薛亚兰 <[email protected]> wrote:
>
> Thanks Thomas, then I knew how to put the custom.js file, and I tried, it
>> worked, Thanks again.
>>
>> however, I have another question about the notebook local directory,
>> what parameter we should set to change it, since in my example, I really
>> have no idea why the local directory of mine should be this value,
>> Serving notebooks from local directory: */tmp/nonroot/jupyter/work*
>> I run the jupyter notebook at this dir */tmp/nonroot/jupyter/,* and I
>> already set the *c.NotebookApp.notebook_dir* to be another value, but
>> the local directory keeps the same, everytime I start a notebook, it is
>> this value
>>
>> */tmp/nonroot/jupyter/work/. *
>> Do you have any idea about these above dir associated issues?
>>
> Setting c.NotebookApp.notebook_dir should indeed set the default notebook
> directory. This will be overridden by command-line arguments, e.g.
>
> jupyter notebook /path/to/notebooks
>
> or
>
> jupyter notebook --notebook-dir /path/to/notebooks
>
> You might verify that this config is picked up, by starting the notebook
> server with --debug.
>
> -MinRK
>
>
>> Thanks in advance!
>>
>> 2016-07-28 4:40 GMT+08:00 Thomas Kluyver <[email protected]>:
>>
>>> On 27 July 2016 at 11:36, <[email protected]> wrote:
>>>
>>>> the config file really produced in MY_CONFIG_DIR, and some files like
>>>> nbserver-20374.json produced in runtime dir of MY_DATA_DIR, but I
>>>> definitely can not find the dirs as custom and files as custom.js, Could
>>>> anyone help me or tell me where are the files' location?
>>>>
>>>
>>> The custom files are not created automatically, but you can put them at
>>> e.g. $JUPYTER_CONFIG_DIR/custom/custom.js
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Project Jupyter" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/jupyter/VF5RHpncN2k/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> [email protected].
>>> To post to this group, send email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/jupyter/CAOvn4qh%2BkSQ_cdGj6M28ZanoXJTNpEgiTrT7AWYc%3D-5XX_anww%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/jupyter/CAOvn4qh%2BkSQ_cdGj6M28ZanoXJTNpEgiTrT7AWYc%3D-5XX_anww%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Yalan Xue ， SIT Lab of Nanjing University
>> Address: Nanjing ,China。
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Project Jupyter" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/CACNymyti%3DfBzxrU56qTk3fMaeFPYit%3DcwQUoYMve2_%2B9VqqCyg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/jupyter/CACNymyti%3DfBzxrU56qTk3fMaeFPYit%3DcwQUoYMve2_%2B9VqqCyg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Project Jupyter" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jupyter/VF5RHpncN2k/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jupyter/CAHNn8BUJ%2BUz-UYqwb4se0Z_Vux08f-d8xLLKRoVw_q9d8DQpZg%40mail.gmail.com
> <https://groups.google.com/d/msgid/jupyter/CAHNn8BUJ%2BUz-UYqwb4se0Z_Vux08f-d8xLLKRoVw_q9d8DQpZg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
Yalan Xue ， SIT Lab of Nanjing University
Address: Nanjing ,China。
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/CACNymysQj1%3DaDTbeoedmYaqsX_sKMdxAKYb1Pbq9MzeUx-Mdig%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
