[jira] [Commented] (SPARK-26134) Upgrading Hadoop to 2.7.4 to fix java.version problem
[ https://issues.apache.org/jira/browse/SPARK-26134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694396#comment-16694396 ] Takanobu Asanuma commented on SPARK-26134: -- I built spark by "build/mvn -DskipTests clean package" and just executed {{bin/spark-shell}} in my local computer. So it used the default version of Hadoop, 2.7.3, which is specified in ./pom.xml. > Upgrading Hadoop to 2.7.4 to fix java.version problem > - > > Key: SPARK-26134 > URL: https://issues.apache.org/jira/browse/SPARK-26134 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Takanobu Asanuma >Priority: Major > > When I ran spark-shell on JDK11+28(2018-09-25), It failed with the error > below. > {noformat} > Exception in thread "main" java.lang.ExceptionInInitializerError > at org.apache.hadoop.util.StringUtils.(StringUtils.java:80) > at > org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273) > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) > at > org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2427) > at scala.Option.getOrElse(Option.scala:121) > at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2427) > at org.apache.spark.SecurityManager.(SecurityManager.scala:79) > at > org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359) > at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:359) > at > org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$9(SparkSubmit.scala:367) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:367) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 > at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) > at java.base/java.lang.String.substring(String.java:1874) > at org.apache.hadoop.util.Shell.(Shell.java:52) > {noformat} > This is a Hadoop issue that fails to parse some {{java.version}}. It has been > fixed from Hadoop-2.7.4(see HADOOP-14586). > Note, Hadoop-2.7.5 or upper have another problem with Spark (SPARK-25330). So > upgrading to 2.7.4 would be fine for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26134) Upgrading Hadoop to 2.7.4 to fix java.version problem
[ https://issues.apache.org/jira/browse/SPARK-26134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694386#comment-16694386 ] Dongjoon Hyun commented on SPARK-26134: --- Thanks. Which Hadoop version are you using then? > Upgrading Hadoop to 2.7.4 to fix java.version problem > - > > Key: SPARK-26134 > URL: https://issues.apache.org/jira/browse/SPARK-26134 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Takanobu Asanuma >Priority: Major > > When I ran spark-shell on JDK11+28(2018-09-25), It failed with the error > below. > {noformat} > Exception in thread "main" java.lang.ExceptionInInitializerError > at org.apache.hadoop.util.StringUtils.(StringUtils.java:80) > at > org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273) > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) > at > org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2427) > at scala.Option.getOrElse(Option.scala:121) > at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2427) > at org.apache.spark.SecurityManager.(SecurityManager.scala:79) > at > org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359) > at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:359) > at > org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$9(SparkSubmit.scala:367) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:367) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 > at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) > at java.base/java.lang.String.substring(String.java:1874) > at org.apache.hadoop.util.Shell.(Shell.java:52) > {noformat} > This is a Hadoop issue that fails to parse some {{java.version}}. It has been > fixed from Hadoop-2.7.4(see HADOOP-14586). > Note, Hadoop-2.7.5 or upper have another problem with Spark (SPARK-25330). So > upgrading to 2.7.4 would be fine for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26134) Upgrading Hadoop to 2.7.4 to fix java.version problem
[ https://issues.apache.org/jira/browse/SPARK-26134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694320#comment-16694320 ] Takanobu Asanuma commented on SPARK-26134: -- Hi, [~dongjoon]. I confirmed that spark-shell failed on jdk-11+28 and passed on jdk-11.0.1+13 with the master branch. I use Oracle OpenJDK, but I could not find the old archive. Seems it still can be downloaded from AdoptOpenJDK. https://adoptopenjdk.net/archive.html?variant=openjdk11=hotspot > Upgrading Hadoop to 2.7.4 to fix java.version problem > - > > Key: SPARK-26134 > URL: https://issues.apache.org/jira/browse/SPARK-26134 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Takanobu Asanuma >Priority: Major > > When I ran spark-shell on JDK11+28(2018-09-25), It failed with the error > below. > {noformat} > Exception in thread "main" java.lang.ExceptionInInitializerError > at org.apache.hadoop.util.StringUtils.(StringUtils.java:80) > at > org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273) > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) > at > org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2427) > at scala.Option.getOrElse(Option.scala:121) > at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2427) > at org.apache.spark.SecurityManager.(SecurityManager.scala:79) > at > org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359) > at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:359) > at > org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$9(SparkSubmit.scala:367) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:367) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 > at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) > at java.base/java.lang.String.substring(String.java:1874) > at org.apache.hadoop.util.Shell.(Shell.java:52) > {noformat} > This is a Hadoop issue that fails to parse some {{java.version}}. It has been > fixed from Hadoop-2.7.4(see HADOOP-14586). > Note, Hadoop-2.7.5 or upper have another problem with Spark (SPARK-25330). So > upgrading to 2.7.4 would be fine for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26134) Upgrading Hadoop to 2.7.4 to fix java.version problem
[ https://issues.apache.org/jira/browse/SPARK-26134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694309#comment-16694309 ] Dongjoon Hyun commented on SPARK-26134: --- Hi, [~tasanuma0829] . Thank you for reporting and sending a PR. If you don't mind, may I ask your environment on JDK 11? > Upgrading Hadoop to 2.7.4 to fix java.version problem > - > > Key: SPARK-26134 > URL: https://issues.apache.org/jira/browse/SPARK-26134 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Takanobu Asanuma >Priority: Major > > When I ran spark-shell on JDK11+28(2018-09-25), It failed with the error > below. > {noformat} > Exception in thread "main" java.lang.ExceptionInInitializerError > at org.apache.hadoop.util.StringUtils.(StringUtils.java:80) > at > org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273) > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) > at > org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2427) > at scala.Option.getOrElse(Option.scala:121) > at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2427) > at org.apache.spark.SecurityManager.(SecurityManager.scala:79) > at > org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359) > at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:359) > at > org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$9(SparkSubmit.scala:367) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:367) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 > at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) > at java.base/java.lang.String.substring(String.java:1874) > at org.apache.hadoop.util.Shell.(Shell.java:52) > {noformat} > This is a Hadoop issue that fails to parse some {{java.version}}. It has been > fixed from Hadoop-2.7.4(see HADOOP-14586). > Note, Hadoop-2.7.5 or upper have another problem with Spark (SPARK-25330). So > upgrading to 2.7.4 would be fine for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26134) Upgrading Hadoop to 2.7.4 to fix java.version problem
[ https://issues.apache.org/jira/browse/SPARK-26134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694289#comment-16694289 ] Apache Spark commented on SPARK-26134: -- User 'tasanuma' has created a pull request for this issue: https://github.com/apache/spark/pull/23101 > Upgrading Hadoop to 2.7.4 to fix java.version problem > - > > Key: SPARK-26134 > URL: https://issues.apache.org/jira/browse/SPARK-26134 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0 >Reporter: Takanobu Asanuma >Priority: Major > > When I ran spark-shell on JDK11+28(2018-09-25), It failed with the error > below. > {noformat} > Exception in thread "main" java.lang.ExceptionInInitializerError > at org.apache.hadoop.util.StringUtils.(StringUtils.java:80) > at > org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273) > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) > at > org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2427) > at scala.Option.getOrElse(Option.scala:121) > at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2427) > at org.apache.spark.SecurityManager.(SecurityManager.scala:79) > at > org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359) > at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:359) > at > org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$9(SparkSubmit.scala:367) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:367) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 > at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) > at java.base/java.lang.String.substring(String.java:1874) > at org.apache.hadoop.util.Shell.(Shell.java:52) > {noformat} > This is a Hadoop issue that fails to parse some {{java.version}}. It has been > fixed from Hadoop-2.7.4(see HADOOP-14586). > Note, Hadoop-2.7.5 or upper have another problem with Spark (SPARK-25330). So > upgrading to 2.7.4 would be fine for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26134) Upgrading Hadoop to 2.7.4 to fix java.version problem
[ https://issues.apache.org/jira/browse/SPARK-26134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694288#comment-16694288 ] Apache Spark commented on SPARK-26134: -- User 'tasanuma' has created a pull request for this issue: https://github.com/apache/spark/pull/23101 > Upgrading Hadoop to 2.7.4 to fix java.version problem > - > > Key: SPARK-26134 > URL: https://issues.apache.org/jira/browse/SPARK-26134 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0 >Reporter: Takanobu Asanuma >Priority: Major > > When I ran spark-shell on JDK11+28(2018-09-25), It failed with the error > below. > {noformat} > Exception in thread "main" java.lang.ExceptionInInitializerError > at org.apache.hadoop.util.StringUtils.(StringUtils.java:80) > at > org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273) > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) > at > org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2427) > at scala.Option.getOrElse(Option.scala:121) > at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2427) > at org.apache.spark.SecurityManager.(SecurityManager.scala:79) > at > org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359) > at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:359) > at > org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$9(SparkSubmit.scala:367) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:367) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 > at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) > at java.base/java.lang.String.substring(String.java:1874) > at org.apache.hadoop.util.Shell.(Shell.java:52) > {noformat} > This is a Hadoop issue that fails to parse some {{java.version}}. It has been > fixed from Hadoop-2.7.4(see HADOOP-14586). > Note, Hadoop-2.7.5 or upper have another problem with Spark (SPARK-25330). So > upgrading to 2.7.4 would be fine for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org