[
https://issues.apache.org/jira/browse/SPARK-20497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987777#comment-15987777
]
Brandon Barker commented on SPARK-20497:
----------------------------------------
Thanks for the quick reply. At the moment, I'm thinking the NPE was due to an
incorrectly configured SparkSession and/or SparkContext, as the SparkSession is
being created by the unofficial package
com.holdenkarau.spark.testing.SparkSessionProvider:
<groupId>com.holdenkarau</groupId>
<artifactId>spark-testing-base_${scala.version.major}</artifactId>
Here's the NPE (line 57 is the val training = ... line mentioned above):
java.lang.NullPointerException
at
edu.cornell.ansci.dairy.econ.util.CsvLookupAnalyzer.<init>(CsvLookupAnalyzer.scala:57)
at
org.cornell.ansci.dairy.econ.util.CsvLookupAnalyzerTest$.setUp(CsvLookupAnalyzerTest.scala:90)
at
org.cornell.ansci.dairy.econ.util.CsvLookupAnalyzerTest.setUp(CsvLookupAnalyzerTest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
When run from the main application where I've configured spark, I get a much
more informative error (aha, a missing "\\" after the C:, oops...)
org.apache.spark.sql.AnalysisException: Path does not exist:
file:/C:Users/brand/Documents/GitHub/sample_linear_regression_data.txt;
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)
...
Fixing this doesn't fix the NPE above when run in the test environment,
indicating it is deep configuration issue, and not an issue with Spark, unless
we could somehow get a "SparkNotConfiguredException" ;). I'll plan to
investigate the testing issue further.
> Unhelpful error messages when trying to load data from file.
> ------------------------------------------------------------
>
> Key: SPARK-20497
> URL: https://issues.apache.org/jira/browse/SPARK-20497
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Brandon Barker
>
> I'm attempting to do the simple task of reproducing the results from the
> linear regression example in Spark. I'm using Windows 10.
> val training = spark.read.format("libsvm")
> .load("C:Users\\brand\\Documents\\GitHub\\sample_linear_regression_data.txt")
> Although the file is definitely at the specified location, I just get a
> java.lang.NullPointerException at this line. The documentation at
> http://spark.apache.org/docs/latest/sql-programming-guide.html#generic-loadsave-functions
> doesn't seem to clear things up. The associated javadocs do do not seem any
> better.
> In my view, such a simple operation should not be troublesome, but perhaps
> I've missed some critical documentation - if so, I apologize.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
