[ https://issues.apache.org/jira/browse/SPARK-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liupengcheng updated SPARK-26529: --------------------------------- Summary: Add debug logs for confArchive when preparing local resource (was: Add logs for IOException when preparing local resource ) > Add debug logs for confArchive when preparing local resource > ------------------------------------------------------------- > > Key: SPARK-26529 > URL: https://issues.apache.org/jira/browse/SPARK-26529 > Project: Spark > Issue Type: Improvement > Components: Deploy, Spark Core > Affects Versions: 2.3.2, 2.4.0 > Reporter: liupengcheng > Priority: Trivial > > Currently, `Client#createConfArchive` do not handle IOException, and some > detail info is not provided in logs. Sometimes, this may delay the time of > locating the root cause of io error. > A case happened in our production environment is that local disk is full, and > the following exception is thrown but no detail path info provided. we have > to investigate all the local disk of the machine to find out the root cause. > {code:java} > Exception in thread "main" java.io.IOException: No space left on device > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:345) > at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253) > at java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:238) > at java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:343) > at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:238) > at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:360) > at org.apache.spark.deploy.yarn.Client.createConfArchive(Client.scala:769) > at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:657) > at > org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:895) > at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:177) > at org.apache.spark.deploy.yarn.Client.run(Client.scala:1202) > at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1261) > at org.apache.spark.deploy.yarn.Client.main(Client.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:767) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:189) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:214) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:128) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > {code} > It make sense for us to catch the IOException and print some useful > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org