Davies Liu created SPARK-11463:
----------------------------------
Summary: SparkContext fail to create in non-main thread in Python
Key: SPARK-11463
URL: https://issues.apache.org/jira/browse/SPARK-11463
Project: Spark
Issue Type: Bug
Reporter: Davies Liu
Priority: Blocker
https://github.com/apache/spark/commit/2e572c4135c3f5ad3061c1f58cdb8a70bed0a9d3#commitcomment-14137386
{code}
Exception in thread Thread-3:
Traceback (most recent call last):
File "/home/asoni/.conda/envs/py35_spark_django/lib/python3.5/threading.py",
line 923, in _bootstrap_inner
self.run()
File "/home/asoni/.conda/envs/py35_spark_django/lib/python3.5/threading.py",
line 871, in run
self._target(*self._args, **self._kwargs)
File "/home/asoni/spark-server/spark_server/spark_jobs/views.py", line 67, in
process_job
from spark_jobs.job_runner import JobRunner
File "/home/asoni/spark-server/spark_server/spark_jobs/job_runner.py", line
2, in <module>
from spark_jobs.extraction_plan_group_runner import *
File
"/home/asoni/spark-server/spark_server/spark_jobs/extraction_plan_group_runner.py",
line 1, in <module>
from spark_jobs.spark_connection import SparkConnection
File "/home/asoni/spark-server/spark_server/spark_jobs/spark_connection.py",
line 5, in <module>
class SparkConnection:
File "/home/asoni/spark-server/spark_server/spark_jobs/spark_connection.py",
line 7, in SparkConnection
sc = SparkContext(conf=conf)
File
"/home/asoni/spark-1.6.0-SNAPSHOT-bin-custom-spark/python/lib/pyspark.zip/pyspark/context.py",
line 114, in __init__
conf, jsc, profiler_cls)
File
"/home/asoni/spark-1.6.0-SNAPSHOT-bin-custom-spark/python/lib/pyspark.zip/pyspark/context.py",
line 225, in _do_init
signal.signal(signal.SIGINT, signal_handler)
File "/home/asoni/.conda/envs/py35_spark_django/lib/python3.5/signal.py",
line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
