[jira] [Comment Edited] (SPARK-13198) sc.stop() does not clean up on driver, causes Java heap OOM.
[
https://issues.apache.org/jira/browse/SPARK-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818498#comment-15818498
]
Dmytro Bielievtsov edited comment on SPARK-13198 at 1/11/17 2:38 PM:
-
[~srowen] Looks like a growing number of people needs this functionality. As
some who knows the codebase, can you give a rough estimate of the amount of
work it might take to make Spark guarantee a good cleanup, equivalent to the
JVM shutdown? Or maybe one could hack this away by somehow restarting the
corresponding JVM without exiting current python interpreter? If this is
reasonable amount of work, I might try to cut out some of our team's time to
work on the corresponding pull request. Thanks!
was (Author: belevtsoff):
[~srowen] Looks like a growing number of people needs this functionality. As
some who knows the codebase, can you give a rough estimate of the amount of
work it might take to make Spark guarantee a good cleanup, equivalent to the
JVM shutdown? Or maybe one could hack this away by somehow restarting the
corresponding JVM without exiting current python interpreter? If this is
reasonable amount of work, I might try to cut out some of our team's time to
work on the corresponding pull request.
> sc.stop() does not clean up on driver, causes Java heap OOM.
>
>
> Key: SPARK-13198
> URL: https://issues.apache.org/jira/browse/SPARK-13198
> Project: Spark
> Issue Type: Bug
> Components: Mesos
>Affects Versions: 1.6.0
>Reporter: Herman Schistad
> Attachments: Screen Shot 2016-02-04 at 16.31.28.png, Screen Shot
> 2016-02-04 at 16.31.40.png, Screen Shot 2016-02-04 at 16.31.51.png, Screen
> Shot 2016-02-08 at 09.30.59.png, Screen Shot 2016-02-08 at 09.31.10.png,
> Screen Shot 2016-02-08 at 10.03.04.png, gc.log
>
>
> When starting and stopping multiple SparkContext's linearly eventually the
> driver stops working with a "io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Java heap space" error.
> Reproduce by running the following code and loading in ~7MB parquet data each
> time. The driver heap space is not changed and thus defaults to 1GB:
> {code:java}
> def main(args: Array[String]) {
> val conf = new SparkConf().setMaster("MASTER_URL").setAppName("")
> conf.set("spark.mesos.coarse", "true")
> conf.set("spark.cores.max", "10")
> for (i <- 1 until 100) {
> val sc = new SparkContext(conf)
> val sqlContext = new SQLContext(sc)
> val events = sqlContext.read.parquet("hdfs://locahost/tmp/something")
> println(s"Context ($i), number of events: " + events.count)
> sc.stop()
> }
> }
> {code}
> The heap space fills up within 20 loops on my cluster. Increasing the number
> of cores to 50 in the above example results in heap space error after 12
> contexts.
> Dumping the heap reveals many equally sized "CoarseMesosSchedulerBackend"
> objects (see attachments). Digging into the inner objects tells me that the
> `executorDataMap` is where 99% of the data in said object is stored. I do
> believe though that this is beside the point as I'd expect this whole object
> to be garbage collected or freed on sc.stop().
> Additionally I can see in the Spark web UI that each time a new context is
> created the number of the "SQL" tab increments by one (i.e. last iteration
> would have SQL99). After doing stop and creating a completely new context I
> was expecting this number to be reset to 1 ("SQL").
> I'm submitting the jar file with `spark-submit` and no special flags. The
> cluster is running Mesos 0.23. I'm running Spark 1.6.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13198) sc.stop() does not clean up on driver, causes Java heap OOM.
[
https://issues.apache.org/jira/browse/SPARK-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818498#comment-15818498
]
Dmytro Bielievtsov commented on SPARK-13198:
[~srowen] Looks like a growing number of people needs this functionality. As
some who knows the codebase, can you give a rough estimate of the amount of
work it might take to make Spark guarantee a good cleanup, equivalent to the
JVM shutdown? Or maybe one could hack this away by somehow restarting the
corresponding JVM without exiting current python interpreter? If this is
reasonable amount of work, I might try to cut out some of our team's time to
work on the corresponding pull request.
> sc.stop() does not clean up on driver, causes Java heap OOM.
>
>
> Key: SPARK-13198
> URL: https://issues.apache.org/jira/browse/SPARK-13198
> Project: Spark
> Issue Type: Bug
> Components: Mesos
>Affects Versions: 1.6.0
>Reporter: Herman Schistad
> Attachments: Screen Shot 2016-02-04 at 16.31.28.png, Screen Shot
> 2016-02-04 at 16.31.40.png, Screen Shot 2016-02-04 at 16.31.51.png, Screen
> Shot 2016-02-08 at 09.30.59.png, Screen Shot 2016-02-08 at 09.31.10.png,
> Screen Shot 2016-02-08 at 10.03.04.png, gc.log
>
>
> When starting and stopping multiple SparkContext's linearly eventually the
> driver stops working with a "io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Java heap space" error.
> Reproduce by running the following code and loading in ~7MB parquet data each
> time. The driver heap space is not changed and thus defaults to 1GB:
> {code:java}
> def main(args: Array[String]) {
> val conf = new SparkConf().setMaster("MASTER_URL").setAppName("")
> conf.set("spark.mesos.coarse", "true")
> conf.set("spark.cores.max", "10")
> for (i <- 1 until 100) {
> val sc = new SparkContext(conf)
> val sqlContext = new SQLContext(sc)
> val events = sqlContext.read.parquet("hdfs://locahost/tmp/something")
> println(s"Context ($i), number of events: " + events.count)
> sc.stop()
> }
> }
> {code}
> The heap space fills up within 20 loops on my cluster. Increasing the number
> of cores to 50 in the above example results in heap space error after 12
> contexts.
> Dumping the heap reveals many equally sized "CoarseMesosSchedulerBackend"
> objects (see attachments). Digging into the inner objects tells me that the
> `executorDataMap` is where 99% of the data in said object is stored. I do
> believe though that this is beside the point as I'd expect this whole object
> to be garbage collected or freed on sc.stop().
> Additionally I can see in the Spark web UI that each time a new context is
> created the number of the "SQL" tab increments by one (i.e. last iteration
> would have SQL99). After doing stop and creating a completely new context I
> was expecting this number to be reset to 1 ("SQL").
> I'm submitting the jar file with `spark-submit` and no special flags. The
> cluster is running Mesos 0.23. I'm running Spark 1.6.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10872) Derby error (XSDB6) when creating new HiveContext after restarting SparkContext
[
https://issues.apache.org/jira/browse/SPARK-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573056#comment-15573056
]
Dmytro Bielievtsov commented on SPARK-10872:
[~sowen] Can you please give me some pointers in the source code so I can start
working towards a pull request?
> Derby error (XSDB6) when creating new HiveContext after restarting
> SparkContext
> ---
>
> Key: SPARK-10872
> URL: https://issues.apache.org/jira/browse/SPARK-10872
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
>Affects Versions: 1.4.0, 1.4.1, 1.5.0
>Reporter: Dmytro Bielievtsov
>
> Starting from spark 1.4.0 (works well on 1.3.1), the following code fails
> with "XSDB6: Another instance of Derby may have already booted the database
> ~/metastore_db":
> {code:python}
> from pyspark import SparkContext, HiveContext
> sc = SparkContext("local[*]", "app1")
> sql = HiveContext(sc)
> sql.createDataFrame([[1]]).collect()
> sc.stop()
> sc = SparkContext("local[*]", "app2")
> sql = HiveContext(sc)
> sql.createDataFrame([[1]]).collect() # Py4J error
> {code}
> This is related to [#SPARK-9539], and I intend to restart spark context
> several times for isolated jobs to prevent cache cluttering and GC errors.
> Here's a larger part of the full error trace:
> {noformat}
> Failed to start database 'metastore_db' with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see
> the next exception for details.
> org.datanucleus.exceptions.NucleusDataStoreException: Failed to start
> database 'metastore_db' with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see
> the next exception for details.
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:516)
> at
> org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:298)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
> at
> org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
> at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:57)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)
> at
> org.apache.hadoop.hive.metastore.Hi
[jira] [Commented] (SPARK-10872) Derby error (XSDB6) when creating new HiveContext after restarting SparkContext
[
https://issues.apache.org/jira/browse/SPARK-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022130#comment-15022130
]
Dmytro Bielievtsov commented on SPARK-10872:
Thanks. I was wondering why is stopping SparkContext outside testing such a bad
idea? For example, one might have a python-based task manager (e.g. luigi) that
wants to run several non-trivial pyspark functions, and still have pythonic
access to the dataframe objects that are passed through the functions (e.g. to
do some common postprocessing). In this case, one has two options:
* have a single SparkContext/HiveContext for the whole pipeline. But this way
Spark sees everything as a single application. Moreover, it's then harder to
clean up cache if one has many intermediate file writes and so forth.
* use spark-submit.sh for each processing task, which adds better encapsulation
at a cost of loosing direct access of the main thread to the data flow and
sometimes ugly workarounds involving pickle to allow for inline code submission.
Therefore, having a fully functioning start/stop syntax seems much cleaner for
this use case. What do you think?
> Derby error (XSDB6) when creating new HiveContext after restarting
> SparkContext
> ---
>
> Key: SPARK-10872
> URL: https://issues.apache.org/jira/browse/SPARK-10872
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
>Affects Versions: 1.4.0, 1.4.1, 1.5.0
>Reporter: Dmytro Bielievtsov
>
> Starting from spark 1.4.0 (works well on 1.3.1), the following code fails
> with "XSDB6: Another instance of Derby may have already booted the database
> ~/metastore_db":
> {code:python}
> from pyspark import SparkContext, HiveContext
> sc = SparkContext("local[*]", "app1")
> sql = HiveContext(sc)
> sql.createDataFrame([[1]]).collect()
> sc.stop()
> sc = SparkContext("local[*]", "app2")
> sql = HiveContext(sc)
> sql.createDataFrame([[1]]).collect() # Py4J error
> {code}
> This is related to [#SPARK-9539], and I intend to restart spark context
> several times for isolated jobs to prevent cache cluttering and GC errors.
> Here's a larger part of the full error trace:
> {noformat}
> Failed to start database 'metastore_db' with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see
> the next exception for details.
> org.datanucleus.exceptions.NucleusDataStoreException: Failed to start
> database 'metastore_db' with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see
> the next exception for details.
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:516)
> at
> org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:298)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
> at
> org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
> at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at
> org.apache.hadoop
[jira] [Commented] (SPARK-10872) Derby error (XSDB6) when creating new HiveContext after restarting SparkContext
[
https://issues.apache.org/jira/browse/SPARK-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022061#comment-15022061
]
Dmytro Bielievtsov commented on SPARK-10872:
Removing {{metastore_db/dbex.lck}} right before {{sc =
SparkContext("local\[*]", "app2")}} precludes the error, but it's a dangerous
workaround. Having something like {{HiveContext.stop()}} that releases the
locks would be best.
> Derby error (XSDB6) when creating new HiveContext after restarting
> SparkContext
> ---
>
> Key: SPARK-10872
> URL: https://issues.apache.org/jira/browse/SPARK-10872
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
>Affects Versions: 1.4.0, 1.4.1, 1.5.0
>Reporter: Dmytro Bielievtsov
>
> Starting from spark 1.4.0 (works well on 1.3.1), the following code fails
> with "XSDB6: Another instance of Derby may have already booted the database
> ~/metastore_db":
> {code:python}
> from pyspark import SparkContext, HiveContext
> sc = SparkContext("local[*]", "app1")
> sql = HiveContext(sc)
> sql.createDataFrame([[1]]).collect()
> sc.stop()
> sc = SparkContext("local[*]", "app2")
> sql = HiveContext(sc)
> sql.createDataFrame([[1]]).collect() # Py4J error
> {code}
> This is related to [#SPARK-9539], and I intend to restart spark context
> several times for isolated jobs to prevent cache cluttering and GC errors.
> Here's a larger part of the full error trace:
> {noformat}
> Failed to start database 'metastore_db' with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see
> the next exception for details.
> org.datanucleus.exceptions.NucleusDataStoreException: Failed to start
> database 'metastore_db' with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see
> the next exception for details.
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:516)
> at
> org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:298)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
> at
> org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
> at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:57)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
> at
> org.apache.hadoop.hive.m
[jira] [Updated] (SPARK-10872) Derby error (XSDB6) when creating new HiveContext after restarting SparkContext
[
https://issues.apache.org/jira/browse/SPARK-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Bielievtsov updated SPARK-10872:
---
Description:
Starting from spark 1.4.0 (works well on 1.3.1), the following code fails with
"XSDB6: Another instance of Derby may have already booted the database
~/metastore_db":
{code:python}
from pyspark import SparkContext, HiveContext
sc = SparkContext("local[*]", "app1")
sql = HiveContext(sc)
sql.createDataFrame([[1]]).collect()
sc.stop()
sc = SparkContext("local[*]", "app2")
sql = HiveContext(sc)
sql.createDataFrame([[1]]).collect() # Py4J error
{code}
This is related to [#SPARK-9539], and I intend to restart spark context several
times for isolated jobs to prevent cache cluttering and GC errors.
Here's a larger part of the full error trace:
{noformat}
Failed to start database 'metastore_db' with class loader
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see the
next exception for details.
org.datanucleus.exceptions.NucleusDataStoreException: Failed to start database
'metastore_db' with class loader
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see the
next exception for details.
at
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:516)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:298)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at
org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:57)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:66)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:199)
at
org.apache.hadoop.hive.ql.metad
[jira] [Created] (SPARK-10872) Derby error (XSDB6) when creating new HiveContext after restarting SparkContext
Dmytro Bielievtsov created SPARK-10872:
--
Summary: Derby error (XSDB6) when creating new HiveContext after
restarting SparkContext
Key: SPARK-10872
URL: https://issues.apache.org/jira/browse/SPARK-10872
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 1.5.0, 1.4.1, 1.4.0
Reporter: Dmytro Bielievtsov
Starting from spark 1.3.1, the following code fails with "XSDB6: Another
instance of Derby may have already booted the database ~/metastore_db":
{code:python}
from pyspark import SparkContext, HiveContext
sc = SparkContext("local[*]", "app1")
sql = HiveContext(sc)
sql.createDataFrame([[1]]).collect()
sc.stop()
sc = SparkContext("local[*]", "app2")
sql = HiveContext(sc)
sql.createDataFrame([[1]]).collect() # Py4J error
{code}
This is related to [#SPARK-9539], and I intend to restart spark context several
times for isolated jobs to prevent cache cluttering and GC errors.
Here's a larger part of the full error trace:
{noformat}
Failed to start database 'metastore_db' with class loader
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see the
next exception for details.
org.datanucleus.exceptions.NucleusDataStoreException: Failed to start database
'metastore_db' with class loader
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@13015ec0, see the
next exception for details.
at
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:516)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:298)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at
org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:57)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:66)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
at
org.apache.hadoo
[jira] [Comment Edited] (SPARK-6282) Strange Python import error when using random() in a lambda function
[
https://issues.apache.org/jira/browse/SPARK-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386564#comment-14386564
]
Dmytro Bielievtsov edited comment on SPARK-6282 at 3/30/15 11:18 AM:
-
I had the same strange issue when computing rdd's via blaze interface
(ContinuumIO). As strangely as it appeared, It disappeared after I removed
'six*' files from '/usr/lib/python2.7/dist-packages' and then reinstalled 'six'
with easy_install-2.7
was (Author: dmytro):
I had the same strange issue when computing rdd's via blaze interface
(ContinuumIO). As strangely as it appeared, It disappeared after I removed
'six*' files from '/usr/lib/python-2.7/dist-packages' and then reinstalled
'six' with easy_install-2.7
> Strange Python import error when using random() in a lambda function
>
>
> Key: SPARK-6282
> URL: https://issues.apache.org/jira/browse/SPARK-6282
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.2.0
> Environment: Kubuntu 14.04, Python 2.7.6
>Reporter: Pavel Laskov
>Priority: Minor
>
> Consider the exemplary Python code below:
>from random import random
>from pyspark.context import SparkContext
>from xval_mllib import read_csv_file_as_list
> if __name__ == "__main__":
> sc = SparkContext(appName="Random() bug test")
> data = sc.parallelize(read_csv_file_as_list('data/malfease-xp.csv'))
> #data = sc.parallelize([1, 2, 3, 4, 5], 2)
> d = data.map(lambda x: (random(), x))
> print d.first()
> Data is read from a large CSV file. Running this code results in a Python
> import error:
> ImportError: No module named _winreg
> If I use 'import random' and 'random.random()' in the lambda function no
> error occurs. Also no error occurs, for both kinds of import statements, for
> a small artificial data set like the one shown in a commented line.
> The full error trace, the source code of csv reading code (function
> 'read_csv_file_as_list' is my own) as well as a sample dataset (the original
> dataset is about 8M large) can be provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-6282) Strange Python import error when using random() in a lambda function
[
https://issues.apache.org/jira/browse/SPARK-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386564#comment-14386564
]
Dmytro Bielievtsov edited comment on SPARK-6282 at 3/30/15 11:18 AM:
-
I had the same strange issue when computing rdd's via blaze interface
(ContinuumIO). As strangely as it appeared, It disappeared after I removed
'six*' files from '/usr/lib/python-2.7/dist-packages' and then reinstalled
'six' with easy_install-2.7
was (Author: dmytro):
I had the same strange issue when computing rdd's via blaze interface
(ContinuumIO). As strangely as it appeared, It strangely disappeared after I
removed 'six*' files from '/usr/lib/python-2.7/dist-packages' and then
reinstalled 'six' with easy_install-2.7
> Strange Python import error when using random() in a lambda function
>
>
> Key: SPARK-6282
> URL: https://issues.apache.org/jira/browse/SPARK-6282
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.2.0
> Environment: Kubuntu 14.04, Python 2.7.6
>Reporter: Pavel Laskov
>Priority: Minor
>
> Consider the exemplary Python code below:
>from random import random
>from pyspark.context import SparkContext
>from xval_mllib import read_csv_file_as_list
> if __name__ == "__main__":
> sc = SparkContext(appName="Random() bug test")
> data = sc.parallelize(read_csv_file_as_list('data/malfease-xp.csv'))
> #data = sc.parallelize([1, 2, 3, 4, 5], 2)
> d = data.map(lambda x: (random(), x))
> print d.first()
> Data is read from a large CSV file. Running this code results in a Python
> import error:
> ImportError: No module named _winreg
> If I use 'import random' and 'random.random()' in the lambda function no
> error occurs. Also no error occurs, for both kinds of import statements, for
> a small artificial data set like the one shown in a commented line.
> The full error trace, the source code of csv reading code (function
> 'read_csv_file_as_list' is my own) as well as a sample dataset (the original
> dataset is about 8M large) can be provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6282) Strange Python import error when using random() in a lambda function
[
https://issues.apache.org/jira/browse/SPARK-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386564#comment-14386564
]
Dmytro Bielievtsov commented on SPARK-6282:
---
I had the same strange issue when computing rdd's via blaze interface
(ContinuumIO). As strangely as it appeared, It strangely disappeared after I
removed 'six*' files from '/usr/lib/python-2.7/dist-packages' and then
reinstalled 'six' with easy_install-2.7
> Strange Python import error when using random() in a lambda function
>
>
> Key: SPARK-6282
> URL: https://issues.apache.org/jira/browse/SPARK-6282
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.2.0
> Environment: Kubuntu 14.04, Python 2.7.6
>Reporter: Pavel Laskov
>Priority: Minor
>
> Consider the exemplary Python code below:
>from random import random
>from pyspark.context import SparkContext
>from xval_mllib import read_csv_file_as_list
> if __name__ == "__main__":
> sc = SparkContext(appName="Random() bug test")
> data = sc.parallelize(read_csv_file_as_list('data/malfease-xp.csv'))
> #data = sc.parallelize([1, 2, 3, 4, 5], 2)
> d = data.map(lambda x: (random(), x))
> print d.first()
> Data is read from a large CSV file. Running this code results in a Python
> import error:
> ImportError: No module named _winreg
> If I use 'import random' and 'random.random()' in the lambda function no
> error occurs. Also no error occurs, for both kinds of import statements, for
> a small artificial data set like the one shown in a commented line.
> The full error trace, the source code of csv reading code (function
> 'read_csv_file_as_list' is my own) as well as a sample dataset (the original
> dataset is about 8M large) can be provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
