[jira] [Comment Edited] (SOLR-9515) Update to Hadoop 3
[ https://issues.apache.org/jira/browse/SOLR-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16762779#comment-16762779 ] Steve Rowe edited comment on SOLR-9515 at 2/7/19 3:44 PM: -- Maven build usage instructions are at {{dev-tools/maven/README.maven}}. FYI currently the Maven Jenkins jobs don't run tests under Maven because nobody maintains that capability - the Maven test runner (surefire) is different from RandomizedRunner, the Ant one (see LUCENE-4045); the Jenkins jobs just build the Maven artifacts and populate the Apache Maven snapshot repo. was (Author: steve_rowe): Maven build usage instructions are at {{dev-tools/maven/README. maven}}. FYI currently the Maven Jenkins jobs don't run tests under Maven because nobody maintains that capability - the Maven test runner (surefire) is different from RandomizedRunner, the Ant one (see LUCENE-4045); the Jenkins jobs just build the Maven artifacts and populate the Apache Maven snapshot repo. > Update to Hadoop 3 > -- > > Key: SOLR-9515 > URL: https://issues.apache.org/jira/browse/SOLR-9515 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Kevin Risden >Priority: Blocker > Fix For: 8.0, 8.x, master (9.0) > > Attachments: SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch, > SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch, > SOLR-9515.patch > > Time Spent: 7h 40m > Remaining Estimate: 0h > > Hadoop 3 is not out yet, but I'd like to iron out the upgrade to be prepared. > I'll start up a dev branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9515) Update to Hadoop 3
[ https://issues.apache.org/jira/browse/SOLR-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757773#comment-16757773 ] Kevin Risden edited comment on SOLR-9515 at 1/31/19 10:40 PM: -- I had tested with just JDK8. This does reproduce locally with JDK11. I checked the failure locally and don't see any symlinks in the path going down. I can revert the master commit since I don't have this working on JDK9+ yet. Sigh. I was going to try a few configs from here: [https://docs.oracle.com/javase/10/security/permissions-jdk1.htm#JSSEC-GUID-83063225-0ACB-4909-9BAB-7F7D4E3749E2] was (Author: risdenk): I had tested with just JDK8. This does reproduce locally though with JDK11. I checked the failure locally and don't see any symlinks in the path going down. I can revert the master commit since I don't have this working on JDK9+ yet. Sigh. I was going to try a few configs from here: https://docs.oracle.com/javase/10/security/permissions-jdk1.htm#JSSEC-GUID-83063225-0ACB-4909-9BAB-7F7D4E3749E2 > Update to Hadoop 3 > -- > > Key: SOLR-9515 > URL: https://issues.apache.org/jira/browse/SOLR-9515 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Kevin Risden >Priority: Blocker > Fix For: 8.0, master (9.0) > > Attachments: SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch, > SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Hadoop 3 is not out yet, but I'd like to iron out the upgrade to be prepared. > I'll start up a dev branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9515) Update to Hadoop 3
[ https://issues.apache.org/jira/browse/SOLR-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757746#comment-16757746 ] Kevin Risden edited comment on SOLR-9515 at 1/31/19 10:08 PM: -- So this caused failures on JDK9+ not sure how the below is possible currently since solr-tests.policy ([https://github.com/apache/lucene-solr/blob/master/lucene/tools/junit4/solr-tests.policy#L27)] allows read access to that path. [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23605] {code:java} [junit4] 2> java.io.IOException: Failed to start sub tasks to add replica in replica map :java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_B3EBC148FC827CD8-001/tempDir-001/hdfsBaseDir/data/data3/current/BP-669531916-88.99.242.108-1548970895105/current/finalized" "read") [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.getVolumeMap(BlockPoolSlice.java:439) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getVolumeMap(FsVolumeImpl.java:1003) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList$1.run(FsVolumeList.java:201) ~[hadoop-hdfs-3.2.0.jar:?] {code} Snippet from solr-tests.policy: {code:java} permission java.io.FilePermission "<>", "read,execute"; permission java.io.FilePermission "${junit4.childvm.cwd}", "read,execute"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp", "read,execute,write,delete"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp${/}-", "read,execute,write,delete"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}jacoco.db", "write"; permission java.io.FilePermission "${junit4.tempDir}${/}*", "read,execute,write,delete"; permission java.io.FilePermission "${clover.db.dir}${/}-", "read,execute,write,delete"; permission java.io.FilePermission "${tests.linedocsfile}", "read"; permission java.nio.file.LinkPermission "hard"; {code} Variables from run: {code:java} junit4.childvm.cwd=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0 junit.tempDir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/temp java.security.manager=org.apache.lucene.util.TestSecurityManager java.security.policy/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/junit4/solr-tests.policy{code} So we should have read on the paths that HDFS is trying to use? was (Author: risdenk): So this caused failures on JDK9+ not sure how the below is possible currently since solr-tests.policy ([https://github.com/apache/lucene-solr/blob/master/lucene/tools/junit4/solr-tests.policy#L27)] allows read access to that path. [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23605] {code:java} [junit4] 2> java.io.IOException: Failed to start sub tasks to add replica in replica map :java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_B3EBC148FC827CD8-001/tempDir-001/hdfsBaseDir/data/data3/current/BP-669531916-88.99.242.108-1548970895105/current/finalized" "read") [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.getVolumeMap(BlockPoolSlice.java:439) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getVolumeMap(FsVolumeImpl.java:1003) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList$1.run(FsVolumeList.java:201) ~[hadoop-hdfs-3.2.0.jar:?] {code} Snippet from solr-tests.policy: {code:java} permission java.io.FilePermission "<>", "read,execute"; permission java.io.FilePermission "${junit4.childvm.cwd}", "read,execute"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp", "read,execute,write,delete"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp${/}-", "read,execute,write,delete"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}jacoco.db", "write"; permission java.io.FilePermission "${junit4.tempDir}${/}*", "read,execute,write,delete"; permission java.io.FilePermission "${clover.db.dir}${/}-", "read,execute,write,delete"; permission java.io.FilePermission "${tests.linedocsfile}", "read"; permission java.nio.file.LinkPermission "hard"; {code} Variables from run: {code:java} junit4.childvm.cwd=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0 junit.tempDir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/temp{code} So we should have rea
[jira] [Comment Edited] (SOLR-9515) Update to Hadoop 3
[ https://issues.apache.org/jira/browse/SOLR-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757746#comment-16757746 ] Kevin Risden edited comment on SOLR-9515 at 1/31/19 9:48 PM: - So this caused failures on JDK9+ not sure how the below is possible currently since solr-tests.policy ([https://github.com/apache/lucene-solr/blob/master/lucene/tools/junit4/solr-tests.policy#L27)] allows read access to that path. [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23605] {code:java} [junit4] 2> java.io.IOException: Failed to start sub tasks to add replica in replica map :java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_B3EBC148FC827CD8-001/tempDir-001/hdfsBaseDir/data/data3/current/BP-669531916-88.99.242.108-1548970895105/current/finalized" "read") [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.getVolumeMap(BlockPoolSlice.java:439) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getVolumeMap(FsVolumeImpl.java:1003) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList$1.run(FsVolumeList.java:201) ~[hadoop-hdfs-3.2.0.jar:?] {code} Snippet from solr-tests.policy: {code:java} permission java.io.FilePermission "<>", "read,execute"; permission java.io.FilePermission "${junit4.childvm.cwd}", "read,execute"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp", "read,execute,write,delete"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp${/}-", "read,execute,write,delete"; permission java.io.FilePermission "${junit4.childvm.cwd}${/}jacoco.db", "write"; permission java.io.FilePermission "${junit4.tempDir}${/}*", "read,execute,write,delete"; permission java.io.FilePermission "${clover.db.dir}${/}-", "read,execute,write,delete"; permission java.io.FilePermission "${tests.linedocsfile}", "read"; permission java.nio.file.LinkPermission "hard"; {code} Variables from run: {code:java} junit4.childvm.cwd=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0 junit.tempDir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/temp{code} So we should have read on the paths that HDFS is trying to use? was (Author: risdenk): So this caused failures on JDK9+ not sure how the below is possible currently since solr-tests.policy allows read access to that path. https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23605 {code:java} [junit4] 2> java.io.IOException: Failed to start sub tasks to add replica in replica map :java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_B3EBC148FC827CD8-001/tempDir-001/hdfsBaseDir/data/data3/current/BP-669531916-88.99.242.108-1548970895105/current/finalized" "read") [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.getVolumeMap(BlockPoolSlice.java:439) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getVolumeMap(FsVolumeImpl.java:1003) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList$1.run(FsVolumeList.java:201) ~[hadoop-hdfs-3.2.0.jar:?] {code} > Update to Hadoop 3 > -- > > Key: SOLR-9515 > URL: https://issues.apache.org/jira/browse/SOLR-9515 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Kevin Risden >Priority: Blocker > Fix For: 8.0, master (9.0) > > Attachments: SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch, > SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Hadoop 3 is not out yet, but I'd like to iron out the upgrade to be prepared. > I'll start up a dev branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9515) Update to Hadoop 3
[ https://issues.apache.org/jira/browse/SOLR-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16756401#comment-16756401 ] Kevin Risden edited comment on SOLR-9515 at 1/30/19 9:13 PM: - The precommit failures from above for Hadoop are real but they are an issue with commons-lang3. I just sent a [message to the commons-lang3 mailing list|http://mail-archives.apache.org/mod_mbox/commons-user/201901.mbox/%3CCAJU9nmhqgzh7VcxyhJNfb4czC2SvJzZd4o6ARcuD4msof1U2Zw%40mail.gmail.com%3E]. I have seen these errors sporadically as well. The stacktrace that is similar is: {code:java} ava.lang.ArrayIndexOutOfBoundsException: 4 [junit4] 2>at org.apache.commons.lang3.time.FastDatePrinter$TextField.appendTo(FastDatePrinter.java:901) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.commons.lang3.time.FastDatePrinter.applyRules(FastDatePrinter.java:573) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.commons.lang3.time.FastDatePrinter.applyRulesToString(FastDatePrinter.java:455) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.commons.lang3.time.FastDatePrinter.format(FastDatePrinter.java:446) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.commons.lang3.time.FastDateFormat.format(FastDateFormat.java:428) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.start(DirectoryScanner.java:281) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.DataNode.initDirectoryScanner(DataNode.java:1090) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1686) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:390) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:819) [hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] {code} This happens when the locale is set. I haven't nailed down which locales all fail but a simple reproducing test case without Hadoop or Lucene/Solr is here: {code:java} long timestamp = System.currentTimeMillis(); Locale.setDefault(Locale.forLanguageTag("ja-JP-u-ca-japanese-x-lvariant-JP")); Assert.assertEquals(SimpleDateFormat.getInstance().format(timestamp), FastDateFormat.getInstance().format(timestamp)); {code} This is with commons-lang3 3.8.1 the latest release. was (Author: risdenk): The precommit failures from above for Hadoop are real but they are an issue with commons-lang3. I just sent a message to the commons-lang3 mailing list. I have seen these errors sporadically as well. The stacktrace that is similar is: {code:java} ava.lang.ArrayIndexOutOfBoundsException: 4 [junit4] 2>at org.apache.commons.lang3.time.FastDatePrinter$TextField.appendTo(FastDatePrinter.java:901) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.commons.lang3.time.FastDatePrinter.applyRules(FastDatePrinter.java:573) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.commons.lang3.time.FastDatePrinter.applyRulesToString(FastDatePrinter.java:455) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.commons.lang3.time.FastDatePrinter.format(FastDatePrinter.java:446) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.commons.lang3.time.FastDateFormat.format(FastDateFormat.java:428) ~[commons-lang3-3.7.jar:3.7] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.start(DirectoryScanner.java:281) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.DataNode.initDirectoryScanner(DataNode.java:1090) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1686) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:390) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:819) [hadoop-hdfs-3.2.0.jar:?] [junit4] 2>at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] {code} This happens when the locale is set. I haven't nailed down which locales all fail but a simple reproducing
[jira] [Comment Edited] (SOLR-9515) Update to Hadoop 3
[ https://issues.apache.org/jira/browse/SOLR-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16756430#comment-16756430 ] Kevin Risden edited comment on SOLR-9515 at 1/30/19 7:03 PM: - [~gerlowskija] there are some more details on the PR in comments about HttpServer2. Near the bottom here: [https://github.com/apache/lucene-solr/pull/553] Gist: Lucene in integration tests spins up a Hadoop cluster which uses HttpServer2 (from Hadoop internally). It only works on Jetty 9.3 but we have Jetty 9.4. So copied/patched HttpServer2 to make the integration tests work. was (Author: risdenk): [~gerlowskija] there are some more details on the PR in comments about HttpServer2. Near the bottom here: [https://github.com/apache/lucene-solr/pull/553] Gist: Lucene in integration tests spins up a Hadoop cluster which HttpServer2 (from Hadoop internally). It only works on Jetty 9.3 but we have Jetty 9.4. So copied/patched HttpServer2 to make the integration tests work. > Update to Hadoop 3 > -- > > Key: SOLR-9515 > URL: https://issues.apache.org/jira/browse/SOLR-9515 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Kevin Risden >Priority: Major > Fix For: 8.0, master (9.0) > > Attachments: SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Hadoop 3 is not out yet, but I'd like to iron out the upgrade to be prepared. > I'll start up a dev branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org