[
https://issues.apache.org/jira/browse/SPARK-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482950#comment-14482950
]
Sean Owen commented on SPARK-6721:
----------------------------------
>From the stack trace, the error is coming from the Mongo driver. That is mild
>evidence that something is wrong with / misconfigured within the driver. It
>looks like it is asserting that a connection is open when it is not. Maybe you
>are reusing a closed connection or using it before it's configured / open?
I usually try to find the source code to see what is going on, like:
http://grepcode.com/file/repo1.maven.org/maven2/org.mongodb/mongo-java-driver/2.13.0/com/mongodb/DBTCPConnector.java/
I also find it's deprecated in the latest code
https://github.com/mongodb/mongo-java-driver/blob/77b7974d8be49c45dcba01c32a8458b121092f87/config/clirr-exclude.yml
Maybe review your code's usage of the Mongo driver in light of the stack trace
and source? If you see a reasonable theory about how Spark is not quite calling
the Hadoop output format correctly (though here, it's just the generic Hadoop
output path, which is used heavily and therefore should be OK), post that here.
Otherwise yeah I suspect you have an issue in your code.
> IllegalStateException when connecting to MongoDB using spark-submit
> -------------------------------------------------------------------
>
> Key: SPARK-6721
> URL: https://issues.apache.org/jira/browse/SPARK-6721
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 1.2.0, 1.2.1, 1.3.0
> Environment: Ubuntu 14.04, Java 8, MongoDB 3.0, Spark 1.3
> Reporter: Luis RodrÃguez Trejo
> Labels: MongoDB, java.lang.IllegalStateexception,
> saveAsNewAPIHadoopFile
>
> I get the following exception when using saveAsNewAPIHadoopFile:
> {code}
> 15/03/23 17:05:34 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 4,
> 10.0.2.15): java.lang.IllegalStateException: open
> at org.bson.util.Assertions.isTrue(Assertions.java:36)
> at com.mongodb.DBTCPConnector.getPrimaryPort(DBTCPConnector.java:406)
> at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:184)
> at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:167)
> at com.mongodb.DBCollection.insert(DBCollection.java:161)
> at com.mongodb.DBCollection.insert(DBCollection.java:107)
> at com.mongodb.DBCollection.save(DBCollection.java:1049)
> at com.mongodb.DBCollection.save(DBCollection.java:1014)
> at
> com.mongodb.hadoop.output.MongoRecordWriter.write(MongoRecordWriter.java:105)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$12.apply(PairRDDFunctions.scala:1000)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$12.apply(PairRDDFunctions.scala:979)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Before Spark 1.3.0 this would result in the application crashing, but now the
> data just remains unprocessed.
> There is no "close" instruction at any part of the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
