[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"
[
https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930094#comment-15930094
]
Jose Soltren commented on SPARK-19982:
--
Pasting in the exception from the link in case the link dies:
Unable to generate an encoder for inner class
`test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without access to
the scope that this class was defined in. Try moving this class out of its
parent class.;
org.apache.spark.sql.AnalysisException: Unable to generate an encoder for inner
class `test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without
access to the scope that this class was defined in. Try moving this class out
of its parent class.;
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$$anonfun$2.applyOrElse(ExpressionEncoder.scala:264)
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$$anonfun$2.applyOrElse(ExpressionEncoder.scala:260)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:243)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:243)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:242)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:248)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:248)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:265)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:305)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:248)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:233)
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.resolve(ExpressionEncoder.scala:260)
at org.apache.spark.sql.Dataset.(Dataset.scala:78)
at org.apache.spark.sql.Dataset.(Dataset.scala:89)
at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:507)
at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:520)
at
test.org.apache.spark.sql.JavaDatasetSuite.testJavaBeanEncoder(JavaDatasetSuite.java:696)
> JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate
> an encoder for inner class"
> -
>
> Key: SPARK-19982
> URL: https://issues.apache.org/jira/browse/SPARK-19982
> Project: Spark
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.1.0
>Reporter: Jose Soltren
> Labels: flaky-test
>
> JavaDatasetSuite.testJavaBeanEncoder fails sporadically with the error below:
> Unable to generate an encoder for inner class
> `test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without access to
> the scope that this class was defined in. Try moving this class out of its
> parent class.
> From https://spark-tests.appspot.com/test-logs/35475788
> [~vanzin] looked into this back in October and reported:
> I ran this test in a loop (both alone and with the rest of the spark-sql
> tests) and never got a failure. I even used the same JDK as Jenkins
> (1.7.0_51).
> Also looked at the code and nothing seems wrong. The errors is when an entry
> with the parent class name is missing from the map kept in OuterScopes.scala,
> but the test populates that map in its first line. So it doesn't look like a
> race nor some issue with weak references (the map uses weak values).
> public void testJavaBeanEncoder() {
> OuterScopes.addOuterScope(this);
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---
[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"
[
https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929296#comment-15929296
]
Wenchen Fan commented on SPARK-19982:
-
yea makes sense, the test harness should hold the {{this}} reference. I have no
idea why this can go wrong, maybe we should just move the test class to
top-level so it's not an inner class anymore.
> JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate
> an encoder for inner class"
> -
>
> Key: SPARK-19982
> URL: https://issues.apache.org/jira/browse/SPARK-19982
> Project: Spark
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.1.0
>Reporter: Jose Soltren
> Labels: flaky-test
>
> JavaDatasetSuite.testJavaBeanEncoder fails sporadically with the error below:
> Unable to generate an encoder for inner class
> `test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without access to
> the scope that this class was defined in. Try moving this class out of its
> parent class.
> From https://spark-tests.appspot.com/test-logs/35475788
> [~vanzin] looked into this back in October and reported:
> I ran this test in a loop (both alone and with the rest of the spark-sql
> tests) and never got a failure. I even used the same JDK as Jenkins
> (1.7.0_51).
> Also looked at the code and nothing seems wrong. The errors is when an entry
> with the parent class name is missing from the map kept in OuterScopes.scala,
> but the test populates that map in its first line. So it doesn't look like a
> race nor some issue with weak references (the map uses weak values).
> public void testJavaBeanEncoder() {
> OuterScopes.addOuterScope(this);
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"
[
https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929275#comment-15929275
]
Michael Armbrust commented on SPARK-19982:
--
I'm not sure if changing weak to strong references will change
[anything|http://stackoverflow.com/questions/299659/what-is-the-difference-between-a-soft-reference-and-a-weak-reference-in-java].
It seems like there must be another handle to {{this}} since the test harness
is actively executing it. So either way it shouldn't be available for garbage
collection, or am I missing something?
> JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate
> an encoder for inner class"
> -
>
> Key: SPARK-19982
> URL: https://issues.apache.org/jira/browse/SPARK-19982
> Project: Spark
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.1.0
>Reporter: Jose Soltren
> Labels: flaky-test
>
> JavaDatasetSuite.testJavaBeanEncoder fails sporadically with the error below:
> Unable to generate an encoder for inner class
> `test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without access to
> the scope that this class was defined in. Try moving this class out of its
> parent class.
> From https://spark-tests.appspot.com/test-logs/35475788
> [~vanzin] looked into this back in October and reported:
> I ran this test in a loop (both alone and with the rest of the spark-sql
> tests) and never got a failure. I even used the same JDK as Jenkins
> (1.7.0_51).
> Also looked at the code and nothing seems wrong. The errors is when an entry
> with the parent class name is missing from the map kept in OuterScopes.scala,
> but the test populates that map in its first line. So it doesn't look like a
> race nor some issue with weak references (the map uses weak values).
> public void testJavaBeanEncoder() {
> OuterScopes.addOuterScope(this);
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"
[
https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929250#comment-15929250
]
Wenchen Fan commented on SPARK-19982:
-
I think this is caused by weak references, a GC may happen right after users
call `OuterScopes.addOuterScope(this);` Shall we use soft reference? cc
[~marmbrus]
> JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate
> an encoder for inner class"
> -
>
> Key: SPARK-19982
> URL: https://issues.apache.org/jira/browse/SPARK-19982
> Project: Spark
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.1.0
>Reporter: Jose Soltren
> Labels: flaky-test
>
> JavaDatasetSuite.testJavaBeanEncoder fails sporadically with the error below:
> Unable to generate an encoder for inner class
> `test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without access to
> the scope that this class was defined in. Try moving this class out of its
> parent class.
> From https://spark-tests.appspot.com/test-logs/35475788
> [~vanzin] looked into this back in October and reported:
> I ran this test in a loop (both alone and with the rest of the spark-sql
> tests) and never got a failure. I even used the same JDK as Jenkins
> (1.7.0_51).
> Also looked at the code and nothing seems wrong. The errors is when an entry
> with the parent class name is missing from the map kept in OuterScopes.scala,
> but the test populates that map in its first line. So it doesn't look like a
> race nor some issue with weak references (the map uses weak values).
> public void testJavaBeanEncoder() {
> OuterScopes.addOuterScope(this);
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"
[
https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928660#comment-15928660
]
Jose Soltren commented on SPARK-19982:
--
[~cloud_fan] added this test as some work related to SPARK-11954. Wenchen, do
you have any thoughts as to why this might be failing intermittently?
Likely not, but I wonder if this has anything to do with outerScopes being a
lazy val in object OuterScopes. Then, possibly, in very rare instances,
Analyzer.scala:ResolveNewInstance could hit the (outer == null) branch and
throw this exception.
We run all the Spark unit tests about a dozen times a night and this has failed
on average twice a month since last May (which is as far back as my data goes).
> JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate
> an encoder for inner class"
> -
>
> Key: SPARK-19982
> URL: https://issues.apache.org/jira/browse/SPARK-19982
> Project: Spark
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.1.0
>Reporter: Jose Soltren
>
> JavaDatasetSuite.testJavaBeanEncoder fails sporadically with the error below:
> Unable to generate an encoder for inner class
> `test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without access to
> the scope that this class was defined in. Try moving this class out of its
> parent class.
> From https://spark-tests.appspot.com/test-logs/35475788
> [~vanzin] looked into this back in October and reported:
> I ran this test in a loop (both alone and with the rest of the spark-sql
> tests) and never got a failure. I even used the same JDK as Jenkins
> (1.7.0_51).
> Also looked at the code and nothing seems wrong. The errors is when an entry
> with the parent class name is missing from the map kept in OuterScopes.scala,
> but the test populates that map in its first line. So it doesn't look like a
> race nor some issue with weak references (the map uses weak values).
> public void testJavaBeanEncoder() {
> OuterScopes.addOuterScope(this);
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
