[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430243#comment-15430243 ] marymwu commented on SPARK-5770: Hey, we have ran into the same issue too. We try to fix this but failed. Anybody can help on this issue, thank so much! > Use addJar() to upload a new jar file to executor, it can't be added to > classloader > --- > > Key: SPARK-5770 > URL: https://issues.apache.org/jira/browse/SPARK-5770 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: meiyoula >Priority: Minor > > First use addJar() to upload a jar to the executor, then change the jar > content and upload it again. We can see the jar file in the local has be > updated, but the classloader still load the old one. The executor log has no > error or exception to point it. > I use spark-shell to test it. And set "spark.files.overwrite" is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367227#comment-15367227 ] Namrata commented on SPARK-5770: Even I ran into the same issue while reloading the jar file using sc.addJar. I am using Job server deployment with long running sparkContext. can we change the Executor ClassLoader implementation to maintain a ClassLoader per jar File and it will help with the reloading of the modified classes. In this way update Jar could be supported but still wont have implementation for delete Jar. > Use addJar() to upload a new jar file to executor, it can't be added to > classloader > --- > > Key: SPARK-5770 > URL: https://issues.apache.org/jira/browse/SPARK-5770 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: meiyoula >Priority: Minor > > First use addJar() to upload a jar to the executor, then change the jar > content and upload it again. We can see the jar file in the local has be > updated, but the classloader still load the old one. The executor log has no > error or exception to point it. > I use spark-shell to test it. And set "spark.files.overwrite" is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347750#comment-15347750 ] huangning commented on SPARK-5770: -- Because it is not support deleteJar or updataJar, so I can't update class without restart thrift server > Use addJar() to upload a new jar file to executor, it can't be added to > classloader > --- > > Key: SPARK-5770 > URL: https://issues.apache.org/jira/browse/SPARK-5770 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: meiyoula >Priority: Minor > > First use addJar() to upload a jar to the executor, then change the jar > content and upload it again. We can see the jar file in the local has be > updated, but the classloader still load the old one. The executor log has no > error or exception to point it. > I use spark-shell to test it. And set "spark.files.overwrite" is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300564#comment-15300564 ] Kai Chen commented on SPARK-5770: - [~pwendell] Sorry for picking up this old thread; but we do have a use case for hot-deployment, which I describe below. We use spark-shell heavily for ad-hoc data analysis as well as iterative development of the analytics code. Commonly the workflow involves the following steps: # Write a small Scala module, and assemble the jar # Start spark-shell with the assembly jar file # Try out some ideas in the shell, then capture the code back into the module # Go back to step 1 and restart the shell This is very similar to what people do in web-app development. And the pain point is similar: in web-app development, a lot of time is spent waiting for new code to be deployed; here, a lot of time is spent waiting for Spark to restart. Having the ability to hot-deploy code in the REPL would help a lot, just as being able to hot-deploy in containers like Play, or using JRebel, has helped boost productivity tremendously. If you like I can create a separate ticket for this. I also have some working code that implements this for 1.5.2, which should be pretty straight-forward to port to the master branch. Cheers! Kai > Use addJar() to upload a new jar file to executor, it can't be added to > classloader > --- > > Key: SPARK-5770 > URL: https://issues.apache.org/jira/browse/SPARK-5770 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: meiyoula >Priority: Minor > > First use addJar() to upload a jar to the executor, then change the jar > content and upload it again. We can see the jar file in the local has be > updated, but the classloader still load the old one. The executor log has no > error or exception to point it. > I use spark-shell to test it. And set "spark.files.overwrite" is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321269#comment-14321269 ] Patrick Wendell commented on SPARK-5770: If the request is to support hot reloading of classes in a live JVM (can't tell if it is), it seems fairly dangerous to do that. I'm not sure how that works if you have live objects of a specific class and then the class definition changes under them. Anyways, as Sean and Marcelo pointed out, it would be helpful to hear what the intended use case is! Use addJar() to upload a new jar file to executor, it can't be added to classloader --- Key: SPARK-5770 URL: https://issues.apache.org/jira/browse/SPARK-5770 Project: Spark Issue Type: Bug Components: Spark Core Reporter: meiyoula First use addJar() to upload a jar to the executor, then change the jar content and upload it again. We can see the jar file in the local has be updated, but the classloader still load the old one. The executor log has no error or exception to point it. I use spark-shell to test it. And set spark.files.overwrite is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320504#comment-14320504 ] Marcelo Vanzin commented on SPARK-5770: --- bq. but the classloader still load the old one. Could you clarify what that means? Due to the way class loading works, if you reference a class that has already been loaded, you won't get the new one, but the one already loaded. Which is one reason why this addJar() can overwrite existing jars functionality is a little sketchy. Use addJar() to upload a new jar file to executor, it can't be added to classloader --- Key: SPARK-5770 URL: https://issues.apache.org/jira/browse/SPARK-5770 Project: Spark Issue Type: Bug Components: Spark Core Reporter: meiyoula First use addJar() to upload a jar to the executor, then change the jar content and upload it again. We can see the jar file in the local has be updated, but the classloader still load the old one. The executor log has no error or exception to point it. I use spark-shell to test it. And set spark.files.overwrite is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320510#comment-14320510 ] Sean Owen commented on SPARK-5770: -- Yeah I think that's the point, that overwriting an existing JAR won't cause any classes to be reloaded, so, should it be an error? or a warning? Use addJar() to upload a new jar file to executor, it can't be added to classloader --- Key: SPARK-5770 URL: https://issues.apache.org/jira/browse/SPARK-5770 Project: Spark Issue Type: Bug Components: Spark Core Reporter: meiyoula First use addJar() to upload a jar to the executor, then change the jar content and upload it again. We can see the jar file in the local has be updated, but the classloader still load the old one. The executor log has no error or exception to point it. I use spark-shell to test it. And set spark.files.overwrite is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320552#comment-14320552 ] Marcelo Vanzin commented on SPARK-5770: --- It might be possible to fix the behavior, although even then the results might be sketchy. Basically, when overwriting jars, you'd have to replace the executor's class loader. That means you need to keep track of the jars added to the class loader, and when adding a new jar, you place it in front of the others and use Thread.currentThread().setContextClassLoader() to replace the class loader. But that's after like 5 seconds of thinking, so there may be a lot of corner cases in doing that. I think the best approach would be to say that overwriting jars is not allowed, even if that doesn't cover all cases. You could still add a different jar that tries to override already loaded classes, and that will have the same confusing effect of the old classes being still used. Use addJar() to upload a new jar file to executor, it can't be added to classloader --- Key: SPARK-5770 URL: https://issues.apache.org/jira/browse/SPARK-5770 Project: Spark Issue Type: Bug Components: Spark Core Reporter: meiyoula First use addJar() to upload a jar to the executor, then change the jar content and upload it again. We can see the jar file in the local has be updated, but the classloader still load the old one. The executor log has no error or exception to point it. I use spark-shell to test it. And set spark.files.overwrite is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318041#comment-14318041 ] Sean Owen commented on SPARK-5770: -- Can you be more specific about where you think the code path doesn't copy the new file? your PR does not touch the copying code but disables overwrite entirely, which is not OK. Use addJar() to upload a new jar file to executor, it can't be added to classloader --- Key: SPARK-5770 URL: https://issues.apache.org/jira/browse/SPARK-5770 Project: Spark Issue Type: Bug Components: Spark Core Reporter: meiyoula First use addJar() to upload a jar to the executor, then change the jar content and upload it again. We can see the jar file in the local has be updated, but the classloader still load the old one. The executor log has no error or exception to point it. I use spark-shell to test it. And set spark.files.overwrite is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318027#comment-14318027 ] Apache Spark commented on SPARK-5770: - User 'XuTingjun' has created a pull request for this issue: https://github.com/apache/spark/pull/4565 Use addJar() to upload a new jar file to executor, it can't be added to classloader --- Key: SPARK-5770 URL: https://issues.apache.org/jira/browse/SPARK-5770 Project: Spark Issue Type: Bug Components: Spark Core Reporter: meiyoula First use addJar() to upload a jar to the executor, then change the jar content and upload it again. We can see the jar file in the local has be updated, but the classloader still load the old one. The executor log has no error or exception to point it. I use spark-shell to test it. And set spark.files.overwrite is true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
