Re: spark.files.userClassPathFirst=true Not Working Correctly
Sorry for the extremely late reply. It turns out that the same error occurred when running on yarn. However, I recently updated my project to depend on cdh5 and the issue I was having disappeared and I am no longer setting the userClassPathFirst to true. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-files-userClassPathFirst-true-Not-Working-Correctly-tp11917p13045.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: spark.files.userClassPathFirst=true Not Working Correctly
Hi, I have Yarn available to test and I'm currently working on getting my application to run correctly on the Yarn cluster. I'll get back to you with the results once I'm able to successfully run it(successfully meaning I at least get to the point where my application currently fails) On Tue, Aug 12, 2014 at 6:46 PM, Marcelo Vanzin [via Apache Spark User List] ml-node+s1001560n12008...@n3.nabble.com wrote: Hi, sorry for the delay. Would you have yarn available to test? Given the discussion in SPARK-2878, this might be a different incarnation of the same underlying issue. The option in Yarn is spark.yarn.user.classpath.first On Mon, Aug 11, 2014 at 1:33 PM, DNoteboom [hidden email] http://user/SendEmail.jtp?type=nodenode=12008i=0 wrote: I'm currently running on my local machine on standalone. The error shows up in my code when I am closing resources using the TaskContext.addOnCompleteCallBack. However, the cause of this error is because of a faulty classLoader which must occur in the Executor in the function createClassLoader. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-files-userClassPathFirst-true-Not-Working-Correctly-tp11917p11921.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: [hidden email] http://user/SendEmail.jtp?type=nodenode=12008i=1 For additional commands, e-mail: [hidden email] http://user/SendEmail.jtp?type=nodenode=12008i=2 -- Marcelo - To unsubscribe, e-mail: [hidden email] http://user/SendEmail.jtp?type=nodenode=12008i=3 For additional commands, e-mail: [hidden email] http://user/SendEmail.jtp?type=nodenode=12008i=4 -- If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/spark-files-userClassPathFirst-true-Not-Working-Correctly-tp11917p12008.html To unsubscribe from spark.files.userClassPathFirst=true Not Working Correctly, click here http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=11917code=ZGFuaWVsQHdpYmlkYXRhLmNvbXwxMTkxN3wyMDg5NTg4OTMz . NAML http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-files-userClassPathFirst-true-Not-Working-Correctly-tp11917p12066.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: spark.files.userClassPathFirst=true Not Working Correctly
Hi, sorry for the delay. Would you have yarn available to test? Given the discussion in SPARK-2878, this might be a different incarnation of the same underlying issue. The option in Yarn is spark.yarn.user.classpath.first On Mon, Aug 11, 2014 at 1:33 PM, DNoteboom dan...@wibidata.com wrote: I'm currently running on my local machine on standalone. The error shows up in my code when I am closing resources using the TaskContext.addOnCompleteCallBack. However, the cause of this error is because of a faulty classLoader which must occur in the Executor in the function createClassLoader. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-files-userClassPathFirst-true-Not-Working-Correctly-tp11917p11921.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: spark.files.userClassPathFirst=true Not Working Correctly
Could you share what's the cluster manager you're using and exactly where the error shows up (driver or executor)? A quick look reveals that Standalone and Yarn use different options to control this, for example. (Maybe that already should be a bug.) On Mon, Aug 11, 2014 at 12:24 PM, DNoteboom dan...@wibidata.com wrote: Currently my code uses commons-pool version 1.6 but Spark uses commons-pool version 1.54. This causes an error when I try to access a method that is visible in 1.6 but not in 1.54. I tried to fix this by setting the userClassPathFirst=true(and I verified that this was set correctly in http://driver:4040 and that my jars are listed in the user-jars). The problem did not go away which means that Spark is not functioning correctly. I added commons-pool-version 1.6 in front of CLASS_PATH in the bin/compute_classpath.sh file and this got rid of my problem, but this is a hacky fix and not a long term solution. I have looked through the Spark source code and it appears to check the URLClassLoader for the user classpath first. I have tried to determine how the user jars are being added to the list of URLs to the classloader with little success. At this point I was trying to debug by installing core-spark so I can edit the source code and then injecting the modified .class files into the Spark-assembly-jar that is in spark-hadoop with little success. Does anyone know why this doesn't work, or have any solutions for this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-files-userClassPathFirst-true-Not-Working-Correctly-tp11917.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: spark.files.userClassPathFirst=true Not Working Correctly
I'm currently running on my local machine on standalone. The error shows up in my code when I am closing resources using the TaskContext.addOnCompleteCallBack. However, the cause of this error is because of a faulty classLoader which must occur in the Executor in the function createClassLoader. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-files-userClassPathFirst-true-Not-Working-Correctly-tp11917p11921.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
