[GitHub] [hadoop] elek commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations
elek commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations URL: https://github.com/apache/hadoop/pull/1582#issuecomment-539368083 If you think about the integration tests, they are independent (I am working to remove that noise). I don't know what has been happened but I think you did some merge instead of rebase. Behind this link: https://github.com/apache/hadoop/pull/1582/files I see hundred of files which are changed but ideally only docker-config files should be there. Try: ``` git checkout HDDS-2217 git rebase origin/trunk git rebase -i trunk # chance all the picks to d before the lines which are not part of your patch git push -f ``` (Or ping me offline if you have problems...) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1612: HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS).
hadoop-yetus commented on issue #1612: HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS). URL: https://github.com/apache/hadoop/pull/1612#issuecomment-539366182 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 2 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 36 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 16 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 62 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 852 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 20 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 955 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 34 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 18 | Maven dependency ordering for patch | | -1 | mvninstall | 38 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 38 | hadoop-ozone in the patch failed. | | -1 | compile | 24 | hadoop-hdds in the patch failed. | | -1 | compile | 18 | hadoop-ozone in the patch failed. | | -1 | javac | 24 | hadoop-hdds in the patch failed. | | -1 | javac | 18 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 55 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 791 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 25 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 34 | hadoop-hdds in the patch failed. | | -1 | findbugs | 20 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 30 | hadoop-hdds in the patch failed. | | -1 | unit | 28 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2498 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1612 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux caa9c2e355c4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4fdf016 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1612/1/artifact/out/patch-co
[jira] [Updated] (HADOOP-16614) Missing leveldbjni package of aarch64 platform
[ https://issues.apache.org/jira/browse/HADOOP-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liusheng updated HADOOP-16614: -- Description: Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of *org.fusesource.leveldbjni* group, but it cannot support ARM platform. see: [https://search.maven.org/search?q=g:org.fusesource.leveldbjni] Because the leveldbjni community is inactivity and the code ([https://github.com/fusesource/leveldbjni]) didn't updated a long time.I will build the leveldbjni package of aarch64 platform, and upload it with other platform packages of *org.fusesource.leveldbjni* to a new *org.openlabtesting.leveldbjni* maven repo. In hadoop code, I will add a new profile aarch64 for for automatically select the *org.openlabtesting.leveldbjni* artifact group and using the aarch64 package of leveldbjni when running on ARM server, this approach has no effect on current code. was: Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of *org.fusesource.leveldbjni* group, but it cannot support ARM platform. see: [https://search.maven.org/search?q=g:org.fusesource.leveldbjni] Because the leveldb code has > Missing leveldbjni package of aarch64 platform > -- > > Key: HADOOP-16614 > URL: https://issues.apache.org/jira/browse/HADOOP-16614 > Project: Hadoop Common > Issue Type: Bug >Reporter: liusheng >Priority: Major > > Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of > *org.fusesource.leveldbjni* group, but it cannot support ARM platform. > see: [https://search.maven.org/search?q=g:org.fusesource.leveldbjni] > Because the leveldbjni community is inactivity and the code > ([https://github.com/fusesource/leveldbjni]) didn't updated a long time.I > will build the leveldbjni package of aarch64 platform, and upload it with > other platform packages of *org.fusesource.leveldbjni* to a new > *org.openlabtesting.leveldbjni* maven repo. In hadoop code, I will add a new > profile aarch64 for for automatically select the > *org.openlabtesting.leveldbjni* artifact group and using the aarch64 package > of leveldbjni when running on ARM server, this approach has no effect on > current code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1606: HDDS-2262. SLEEP_SECONDS: command not found
adoroszlai commented on issue #1606: HDDS-2262. SLEEP_SECONDS: command not found URL: https://github.com/apache/hadoop/pull/1606#issuecomment-539363460 Thanks @anuengineer for reviewing and committing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1605: HDDS-2259. Container Data Scrubber computes wrong checksum
adoroszlai commented on issue #1605: HDDS-2259. Container Data Scrubber computes wrong checksum URL: https://github.com/apache/hadoop/pull/1605#issuecomment-539363289 Thanks @anuengineer for reviewing and committing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1607: HDDS-2264. Improve output of TestOzoneContainer
adoroszlai commented on issue #1607: HDDS-2264. Improve output of TestOzoneContainer URL: https://github.com/apache/hadoop/pull/1607#issuecomment-539362735 Thanks @anuengineer for reviewing and committing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
adoroszlai commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590#issuecomment-539362568 Thanks @anuengineer for reviewing and committing it. The `` gets rid of warnings about using raw type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16614) Missing leveldbjni package of aarch64 platform
[ https://issues.apache.org/jira/browse/HADOOP-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liusheng updated HADOOP-16614: -- Description: Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of *org.fusesource.leveldbjni* group, but it cannot support ARM platform. see: [https://search.maven.org/search?q=g:org.fusesource.leveldbjni] Because the leveldb code has was: Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of *org.fusesource.leveldbjni* group, but it cannot support ARM platform. see: https://search.maven.org/search?q=g:org.fusesource.leveldbjni > Missing leveldbjni package of aarch64 platform > -- > > Key: HADOOP-16614 > URL: https://issues.apache.org/jira/browse/HADOOP-16614 > Project: Hadoop Common > Issue Type: Bug >Reporter: liusheng >Priority: Major > > Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of > *org.fusesource.leveldbjni* group, but it cannot support ARM platform. > see: [https://search.maven.org/search?q=g:org.fusesource.leveldbjni] > Because the leveldb code has -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16492) Support HuaweiCloud Object Storage - as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjun updated HADOOP-16492: -- Description: Added support for HuaweiCloud OBS([https://www.huaweicloud.com/en-us/product/obs.html]) to Hadoop, just like what we do before for S3, ADL, OSS, etc. With simple configuration, Hadoop applications can read/write data from OBS without any code change. was: Added support for HuaweiCloud OBS([https://www.huaweicloud.com/en-us/product/obs.html]) to Hadoop, just like what we do before for S3, ADL, OSS, etc. With simple configuration, Hadoop applications can read/write data from OBS without any code change. h2. > Support HuaweiCloud Object Storage - as a file system in Hadoop > --- > > Key: HADOOP-16492 > URL: https://issues.apache.org/jira/browse/HADOOP-16492 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 3.3.0 >Reporter: zhongjun >Priority: Major > Attachments: huaweicloud-obs-integrate.pdf > > > Added support for HuaweiCloud > OBS([https://www.huaweicloud.com/en-us/product/obs.html]) to Hadoop, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from OBS without any code change. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on issue #1612: HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS).
swagle commented on issue #1612: HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS). URL: https://github.com/apache/hadoop/pull/1612#issuecomment-539350641 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle opened a new pull request #1612: HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS).
swagle opened a new pull request #1612: HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS). URL: https://github.com/apache/hadoop/pull/1612 Fixed all the places where the parameter would need evaluation, left on only string literals and pre-evaluated variables intact. Addressed LOG.debug, LOGGER.debug, LOG.trace and LOGGE.trace by search and edit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16641) RPC: Heavy contention on Configuration.getClassByNameOrNull
[ https://issues.apache.org/jira/browse/HADOOP-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HADOOP-16641: -- Attachment: llap-rpc-locks.svg > RPC: Heavy contention on Configuration.getClassByNameOrNull > > > Key: HADOOP-16641 > URL: https://issues.apache.org/jira/browse/HADOOP-16641 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal Vijayaraghavan >Priority: Major > Labels: performance > Attachments: config-get-class-by-name.png, llap-rpc-locks.svg > > > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589 > {code} > map = Collections.synchronizedMap( > new WeakHashMap>>()); > {code} > This synchronizes all lookups across the same class-loader across all threads > & yields rpc threads. > !config-get-class-by-name.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16641) RPC: Heavy contention on Configuration.getClassByNameOrNull
[ https://issues.apache.org/jira/browse/HADOOP-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HADOOP-16641: -- Description: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589 {code} map = Collections.synchronizedMap( new WeakHashMap>>()); {code} This synchronizes all lookups across the same class-loader across all threads & yields rpc threads. !config-get-class-by-name.png! When reading from HDFS with good locality, this fills up the contended lock profile with almost no other contributors to the locking - see [^llap-rpc-locks.svg] was: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589 {code} map = Collections.synchronizedMap( new WeakHashMap>>()); {code} This synchronizes all lookups across the same class-loader across all threads & yields rpc threads. !config-get-class-by-name.png! > RPC: Heavy contention on Configuration.getClassByNameOrNull > > > Key: HADOOP-16641 > URL: https://issues.apache.org/jira/browse/HADOOP-16641 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal Vijayaraghavan >Priority: Major > Labels: performance > Attachments: config-get-class-by-name.png, llap-rpc-locks.svg > > > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589 > {code} > map = Collections.synchronizedMap( > new WeakHashMap>>()); > {code} > This synchronizes all lookups across the same class-loader across all threads > & yields rpc threads. > !config-get-class-by-name.png! > When reading from HDFS with good locality, this fills up the contended lock > profile with almost no other contributors to the locking - see > [^llap-rpc-locks.svg] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] christeoh commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations
christeoh commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations URL: https://github.com/apache/hadoop/pull/1582#issuecomment-539341389 Looks like there's some issues with Ratis? Is this related to the patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16641) RPC: Heavy contention on Configuration.getClassByNameOrNull
[ https://issues.apache.org/jira/browse/HADOOP-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HADOOP-16641: -- Labels: performance (was: ) > RPC: Heavy contention on Configuration.getClassByNameOrNull > > > Key: HADOOP-16641 > URL: https://issues.apache.org/jira/browse/HADOOP-16641 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal Vijayaraghavan >Priority: Major > Labels: performance > Attachments: config-get-class-by-name.png > > > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589 > {code} > map = Collections.synchronizedMap( > new WeakHashMap>>()); > {code} > This synchronizes all lookups across the same class-loader across all threads > & yields rpc threads. > !config-get-class-by-name.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16641) RPC: Heavy contention on Configuration.getClassByNameOrNull
[ https://issues.apache.org/jira/browse/HADOOP-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HADOOP-16641: -- Component/s: conf > RPC: Heavy contention on Configuration.getClassByNameOrNull > > > Key: HADOOP-16641 > URL: https://issues.apache.org/jira/browse/HADOOP-16641 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal Vijayaraghavan >Priority: Major > Attachments: config-get-class-by-name.png > > > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589 > {code} > map = Collections.synchronizedMap( > new WeakHashMap>>()); > {code} > This synchronizes all lookups across the same class-loader across all threads > & yields rpc threads. > !config-get-class-by-name.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16641) RPC: Heavy contention on Configuration.getClassByNameOrNull
Gopal Vijayaraghavan created HADOOP-16641: - Summary: RPC: Heavy contention on Configuration.getClassByNameOrNull Key: HADOOP-16641 URL: https://issues.apache.org/jira/browse/HADOOP-16641 Project: Hadoop Common Issue Type: Bug Reporter: Gopal Vijayaraghavan Attachments: config-get-class-by-name.png https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589 {code} map = Collections.synchronizedMap( new WeakHashMap>>()); {code} This synchronizes all lookups across the same class-loader across all threads & yields rpc threads. !config-get-class-by-name.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jeffsaremi commented on issue #1480: HDFS-14857. FS operations fail in HA mode: DataNode fails to connect to NameNode
jeffsaremi commented on issue #1480: HDFS-14857. FS operations fail in HA mode: DataNode fails to connect to NameNode URL: https://github.com/apache/hadoop/pull/1480#issuecomment-539308184 I went back and revised some of the code. It looked like we're trying to do some really important stuff right inside the perfromFailover() method. This didn't feel right and we didn't want to deal with all sorts of things that could go wrong. Instead we're doing the minimal here -- making sure the proxy is nullified -- and then when the Retry loop calls the getProxy() we deal with actual IP resetting. This was important since we already had a lazy initializer inside AbstractNNFailoverProxyProvider and we could just reuse that . Also there seemed to be some overlap in the old way of resettingProxy and what AbstractNNFailoverProxyProvider is doing. So we're avoiding too much redundancy. Inside AbstractNNFailoverProxyProvider there are actually two spots that this could happen: createProxyIfNeeded() and getProxyAddresses() either one could be altered to handle the IP address reset as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Cosss7 closed pull request #1129: HDFS-14509 DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x
Cosss7 closed pull request #1129: HDFS-14509 DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x URL: https://github.com/apache/hadoop/pull/1129 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ChenSammi commented on a change in pull request #1469: HDDS-2034. Async RATIS pipeline creation and destroy through heartbea…
ChenSammi commented on a change in pull request #1469: HDDS-2034. Async RATIS pipeline creation and destroy through heartbea… URL: https://github.com/apache/hadoop/pull/1469#discussion_r332321239 ## File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CreatePipelineCommandHandler.java ## @@ -0,0 +1,226 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.ozone.container.common.statemachine.commandhandler; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdds.protocol.DatanodeDetails; +import org.apache.hadoop.hdds.protocol.proto. +StorageContainerDatanodeProtocolProtos.CreatePipelineCommandProto; +import org.apache.hadoop.hdds.protocol.proto. +StorageContainerDatanodeProtocolProtos.SCMCommandProto; +import org.apache.hadoop.hdds.protocol.proto. +StorageContainerDatanodeProtocolProtos.CreatePipelineACKProto; +import org.apache.hadoop.hdds.ratis.RatisHelper; +import org.apache.hadoop.hdds.scm.ScmConfigKeys; +import org.apache.hadoop.hdds.scm.pipeline.PipelineID; +import org.apache.hadoop.hdds.security.x509.SecurityConfig; +import org.apache.hadoop.io.MultipleIOException; +import org.apache.hadoop.ozone.container.common.statemachine +.SCMConnectionManager; +import org.apache.hadoop.ozone.container.common.statemachine.StateContext; +import org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer; +import org.apache.hadoop.ozone.protocol.commands.CommandStatus; +import org.apache.hadoop.ozone.protocol.commands.CreatePipelineCommand; +import org.apache.hadoop.ozone.protocol.commands.CreatePipelineCommandStatus; +import org.apache.hadoop.ozone.protocol.commands.SCMCommand; +import org.apache.hadoop.util.Time; +import org.apache.ratis.client.RaftClient; +import org.apache.ratis.grpc.GrpcTlsConfig; +import org.apache.ratis.protocol.NotLeaderException; +import org.apache.ratis.protocol.RaftClientReply; +import org.apache.ratis.protocol.RaftGroup; +import org.apache.ratis.protocol.RaftGroupId; +import org.apache.ratis.protocol.RaftPeer; +import org.apache.ratis.retry.RetryPolicy; +import org.apache.ratis.rpc.SupportedRpcType; +import org.apache.ratis.util.TimeDuration; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.RejectedExecutionException; +import java.util.function.Consumer; +import java.util.stream.Collectors; + +/** + * Handler for create pipeline command received from SCM. + */ +public class CreatePipelineCommandHandler implements CommandHandler { + + private static final Logger LOG = + LoggerFactory.getLogger(CreatePipelineCommandHandler.class); + + private int invocationCount; + private long totalTime; + + /** + * Constructs a createPipelineCommand handler. + */ + public CreatePipelineCommandHandler() { + } + + /** + * Handles a given SCM command. + * + * @param command - SCM Command + * @param ozoneContainer- Ozone Container. + * @param context - Current Context. + * @param connectionManager - The SCMs that we are talking to. + */ + @Override + public void handle(SCMCommand command, OzoneContainer ozoneContainer, + StateContext context, SCMConnectionManager connectionManager) { +invocationCount++; +final long startTime = Time.monotonicNow(); +final DatanodeDetails dn = context.getParent() +.getDatanodeDetails(); +final CreatePipelineCommandProto createCommand = +((CreatePipelineCommand)command).getProto(); +final PipelineID pipelineID = PipelineID.getFromProtobuf( +createCommand.getPipelineID()); +Collection peers = +createCommand.getDatanodeList().stream() +.map(DatanodeDetails::getFromProtoBuf) +.collect(Collectors.toList()); + +final CreatePipelineACKProto createPipelineACK = +CreatePipelineACKProto.newBuilder() +.setPipelineID(createCommand.getPipelineID()) +.setDatanodeUUID(dn.getUuidString()).build(); +boolean success = fal
[GitHub] [hadoop] ChenSammi commented on issue #1469: HDDS-2034. Async RATIS pipeline creation and destroy through heartbea…
ChenSammi commented on issue #1469: HDDS-2034. Async RATIS pipeline creation and destroy through heartbea… URL: https://github.com/apache/hadoop/pull/1469#issuecomment-539294113 > > I think the purpose of safemode is to guarantee that Ozone cluster is ready to provide service to Ozone client once safemode is exited. > > @ChenSammi I agree with that. I think the problem occurs with OneReplicaPipelineSafeModeRule. This rule makes sure that atleast one datanode in the old pipeline is reported so that reads for OPEN containers can go through. Here I think that old pipelines need to be tracked separately. OK, I will try to separate the olds from the new ones. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16614) Missing leveldbjni package of aarch64 platform
[ https://issues.apache.org/jira/browse/HADOOP-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liusheng updated HADOOP-16614: -- Description: Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of *org.fusesource.leveldbjni* group, but it cannot support ARM platform. see: https://search.maven.org/search?q=g:org.fusesource.leveldbjni was:Currently, Hadoop denpend on the *leveldbjni-all:1.8* package, but the ARM platform cannot be supported. > Missing leveldbjni package of aarch64 platform > -- > > Key: HADOOP-16614 > URL: https://issues.apache.org/jira/browse/HADOOP-16614 > Project: Hadoop Common > Issue Type: Bug >Reporter: liusheng >Priority: Major > > Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of > *org.fusesource.leveldbjni* group, but it cannot support ARM platform. > see: https://search.maven.org/search?q=g:org.fusesource.leveldbjni -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946430#comment-16946430 ] Wei-Chiu Chuang commented on HADOOP-16152: -- The compilation error doesn't look related. See HDFS-14900 > Upgrade Eclipse Jetty version to 9.4.x > -- > > Key: HADOOP-16152 > URL: https://issues.apache.org/jira/browse/HADOOP-16152 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Yuming Wang >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, > HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.005.patch, > HADOOP-16152.006.patch, HADOOP-16152.v1.patch > > > Some big data projects have been upgraded Jetty to 9.4.x, which causes some > compatibility issues. > Spark: > [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146] > Calcite: > [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87] > Hive: HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations
hadoop-yetus commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations URL: https://github.com/apache/hadoop/pull/1582#issuecomment-539289204 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 10 | https://github.com/apache/hadoop/pull/1582 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1582 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/3/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] christeoh commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations
christeoh commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations URL: https://github.com/apache/hadoop/pull/1582#issuecomment-539286942 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1588: HDDS-1986. Fix listkeys API.
anuengineer commented on a change in pull request #1588: HDDS-1986. Fix listkeys API. URL: https://github.com/apache/hadoop/pull/1588#discussion_r332314620 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -680,26 +688,85 @@ public boolean isBucketEmpty(String volume, String bucket) seekPrefix = getBucketKey(volumeName, bucketName + OM_KEY_PREFIX); } int currentCount = 0; -try (TableIterator> keyIter = -getKeyTable() -.iterator()) { - KeyValue kv = keyIter.seek(seekKey); - while (currentCount < maxKeys && keyIter.hasNext()) { -kv = keyIter.next(); -// Skip the Start key if needed. -if (kv != null && skipStartKey && kv.getKey().equals(seekKey)) { - continue; + + +TreeMap cacheKeyMap = new TreeMap<>(); +Set deletedKeySet = new TreeSet<>(); +Iterator, CacheValue>> iterator = +keyTable.cacheIterator(); + +//TODO: We can avoid this iteration if table cache has stored entries in +// treemap. Currently HashMap is used in Cache. HashMap get operation is an +// constant time operation, where as for treeMap get is log(n). +// So if we move to treemap, the get operation will be affected. As get +// is frequent operation on table. So, for now in list we iterate cache map +// and construct treeMap which match with keyPrefix and are greater than or +// equal to startKey. Later we can revisit this, if list operation +// is becoming slow. +while (iterator.hasNext()) { Review comment: I am ok with putting this change in if we can prove that we can do large list keys. You might want to borrow the DB from @nandakumar131 and see if you can list keys with this patch, just a thought. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1588: HDDS-1986. Fix listkeys API.
anuengineer commented on a change in pull request #1588: HDDS-1986. Fix listkeys API. URL: https://github.com/apache/hadoop/pull/1588#discussion_r332255281 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -645,7 +648,12 @@ public boolean isBucketEmpty(String volume, String bucket) @Override public List listKeys(String volumeName, String bucketName, String startKey, String keyPrefix, int maxKeys) throws IOException { + List result = new ArrayList<>(); +if (maxKeys == 0) { Review comment: or <= 0 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1588: HDDS-1986. Fix listkeys API.
anuengineer commented on a change in pull request #1588: HDDS-1986. Fix listkeys API. URL: https://github.com/apache/hadoop/pull/1588#discussion_r332255873 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -680,26 +688,85 @@ public boolean isBucketEmpty(String volume, String bucket) seekPrefix = getBucketKey(volumeName, bucketName + OM_KEY_PREFIX); } int currentCount = 0; -try (TableIterator> keyIter = -getKeyTable() -.iterator()) { - KeyValue kv = keyIter.seek(seekKey); - while (currentCount < maxKeys && keyIter.hasNext()) { -kv = keyIter.next(); -// Skip the Start key if needed. -if (kv != null && skipStartKey && kv.getKey().equals(seekKey)) { - continue; + + +TreeMap cacheKeyMap = new TreeMap<>(); +Set deletedKeySet = new TreeSet<>(); +Iterator, CacheValue>> iterator = +keyTable.cacheIterator(); + +//TODO: We can avoid this iteration if table cache has stored entries in +// treemap. Currently HashMap is used in Cache. HashMap get operation is an +// constant time operation, where as for treeMap get is log(n). +// So if we move to treemap, the get operation will be affected. As get +// is frequent operation on table. So, for now in list we iterate cache map +// and construct treeMap which match with keyPrefix and are greater than or +// equal to startKey. Later we can revisit this, if list operation +// is becoming slow. +while (iterator.hasNext()) { Review comment: How many keys are expected in this cache? and how many in the tree ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1588: HDDS-1986. Fix listkeys API.
anuengineer commented on a change in pull request #1588: HDDS-1986. Fix listkeys API. URL: https://github.com/apache/hadoop/pull/1588#discussion_r332256331 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -680,26 +688,85 @@ public boolean isBucketEmpty(String volume, String bucket) seekPrefix = getBucketKey(volumeName, bucketName + OM_KEY_PREFIX); } int currentCount = 0; -try (TableIterator> keyIter = -getKeyTable() -.iterator()) { - KeyValue kv = keyIter.seek(seekKey); - while (currentCount < maxKeys && keyIter.hasNext()) { -kv = keyIter.next(); -// Skip the Start key if needed. -if (kv != null && skipStartKey && kv.getKey().equals(seekKey)) { - continue; + + +TreeMap cacheKeyMap = new TreeMap<>(); +Set deletedKeySet = new TreeSet<>(); +Iterator, CacheValue>> iterator = +keyTable.cacheIterator(); + +//TODO: We can avoid this iteration if table cache has stored entries in +// treemap. Currently HashMap is used in Cache. HashMap get operation is an +// constant time operation, where as for treeMap get is log(n). +// So if we move to treemap, the get operation will be affected. As get +// is frequent operation on table. So, for now in list we iterate cache map +// and construct treeMap which match with keyPrefix and are greater than or +// equal to startKey. Later we can revisit this, if list operation +// is becoming slow. +while (iterator.hasNext()) { Review comment: I feel that we are better off leaving the old code in place...where we can read from the DB.. Worst, we might have to make sure that cache is flushed to DB before doing the list operation.But practically it may not matter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
anuengineer commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332313947 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String bucket) } int currentCount = 0; -try (TableIterator> -bucketIter = bucketTable.iterator()) { - KeyValue kv = bucketIter.seek(startKey); - while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) { -kv = bucketIter.next(); -// Skip the Start Bucket if needed. -if (kv != null && skipStartKey && -kv.getKey().equals(startKey)) { + +// For Bucket it is full cache, so we can just iterate in-memory table +// cache. +Iterator, CacheValue>> iterator = Review comment: You can consider this comment resolved. Thanks for the explanation. I am leaving it open for other reviewers who might want to read this patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
anuengineer commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332313824 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String bucket) } int currentCount = 0; -try (TableIterator> -bucketIter = bucketTable.iterator()) { - KeyValue kv = bucketIter.seek(startKey); - while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) { -kv = bucketIter.next(); -// Skip the Start Bucket if needed. -if (kv != null && skipStartKey && -kv.getKey().equals(startKey)) { + +// For Bucket it is full cache, so we can just iterate in-memory table +// cache. +Iterator, CacheValue>> iterator = Review comment: > But when we want to list all buckets is /vol2, we will iterate the entries from start, and reach to /vol2 in the cache and once the maximum count is reached we return from there. The architecture of SkipList prevents us from iterating all the keys. That is good enough, I was worried that we will walk all the entries. I missed we were using a skipList based hash table. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
arp7 commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#discussion_r332313554 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java ## @@ -105,15 +109,66 @@ public OzoneManagerLock(Configuration conf) { * should be bucket name. For remaining all resource only one param should * be passed. */ + @Deprecated public boolean acquireLock(Resource resource, String... resources) { String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + /** + * Acquire read lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread Review comment: Why is that? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
arp7 commented on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#issuecomment-539279865 I looked through the JDK implementation of read-write locks a couple of years ago. Even in non-fair mode there is prevention against starvation. HDFS uses non-fair mode by default and works well even for very busy Name Nodes. However we can make the lock fair for now, and evaluate making it non-fair later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
arp7 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332306087 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/TableCacheImpl.java ## @@ -47,15 +48,22 @@ public class TableCacheImpl implements TableCache { - private final ConcurrentHashMap cache; + private final Map cache; private final NavigableSet> epochEntries; private ExecutorService executorService; private CacheCleanupPolicy cleanupPolicy; public TableCacheImpl(CacheCleanupPolicy cleanupPolicy) { -cache = new ConcurrentHashMap<>(); + +// As for full table cache only we need elements to be inserted in sorted +// manner, so that list will be easy. For other we can go with Hash map. +if (cleanupPolicy == CacheCleanupPolicy.NEVER) { Review comment: We do need a solution for this, it's not very good to keep adding special checks for Cache policy NEVER. The code will become fragile. I think when we first added cache policies you proposed making it an interface. Let's do that now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
arp7 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332305881 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java ## @@ -53,4 +53,18 @@ public boolean equals(Object o) { public int hashCode() { return Objects.hash(key); } + + @Override + public int compareTo(Object o) { +if(Objects.equals(key, ((CacheKey)o).key)) { + return 0; +} else { + if (key instanceof String) { +return ((String) key).compareTo((String) ((CacheKey)o).key); + } else { +// If not type string, convert to string and compare. +return key.toString().compareToCacheKey) o).key).toString()); Review comment: I think the correct fix is to make OzoneTokenIdentifier a Comparable, and then enforce that the CacheKey implements Comparable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16640) WASB: Override getCanonicalServiceName() to return full url of WASB filesystem
Da Zhou created HADOOP-16640: Summary: WASB: Override getCanonicalServiceName() to return full url of WASB filesystem Key: HADOOP-16640 URL: https://issues.apache.org/jira/browse/HADOOP-16640 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.2 Reporter: Da Zhou HBase calls getCanonicalServiceName() to check if two FS are the same: [https://github.com/apache/hbase/blob/10180e232ebf886c9577d77eb91ce64b51564dfc/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L117] This is creating some issues for customer because the current WASB relied on the default implementation of getCanonicalServiceName() and will return "ip:port". Will override getCanonicalServiceName() in WASB to return the full URI of the fs, and this would be configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16640) WASB: Override getCanonicalServiceName() to return full url of WASB filesystem
[ https://issues.apache.org/jira/browse/HADOOP-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou reassigned HADOOP-16640: Assignee: Da Zhou > WASB: Override getCanonicalServiceName() to return full url of WASB filesystem > -- > > Key: HADOOP-16640 > URL: https://issues.apache.org/jira/browse/HADOOP-16640 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.2 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > > HBase calls getCanonicalServiceName() to check if two FS are the same: > [https://github.com/apache/hbase/blob/10180e232ebf886c9577d77eb91ce64b51564dfc/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L117] > This is creating some issues for customer because the current WASB relied on > the default implementation of getCanonicalServiceName() and will return > "ip:port". > Will override getCanonicalServiceName() in WASB to return the full URI of > the fs, and this would be configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #1588: HDDS-1986. Fix listkeys API.
arp7 commented on a change in pull request #1588: HDDS-1986. Fix listkeys API. URL: https://github.com/apache/hadoop/pull/1588#discussion_r332304634 ## File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java ## @@ -0,0 +1,298 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ + +package org.apache.hadoop.ozone.om; +import com.google.common.base.Optional; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.hdds.protocol.StorageType; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.utils.db.cache.CacheKey; +import org.apache.hadoop.hdds.utils.db.cache.CacheValue; +import org.apache.hadoop.ozone.om.helpers.OmBucketInfo; +import org.apache.hadoop.ozone.om.helpers.OmKeyInfo; +import org.apache.hadoop.ozone.om.request.TestOMRequestUtils; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.util.List; +import java.util.TreeSet; + +import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS; + +/** + * Tests OzoneManager MetadataManager. + */ +public class TestOmMetadataManager { + + private OMMetadataManager omMetadataManager; + private OzoneConfiguration ozoneConfiguration; + + @Rule + public TemporaryFolder folder = new TemporaryFolder(); + + + @Before + public void setup() throws Exception { +ozoneConfiguration = new OzoneConfiguration(); +ozoneConfiguration.set(OZONE_OM_DB_DIRS, +folder.getRoot().getAbsolutePath()); +omMetadataManager = new OmMetadataManagerImpl(ozoneConfiguration); + } + @Test + public void testListKeys() throws Exception { + +String volumeNameA = "volumeA"; +String volumeNameB = "volumeB"; +String ozoneBucket = "ozoneBucket"; +String hadoopBucket = "hadoopBucket"; + + +// Create volumes and buckets. +TestOMRequestUtils.addVolumeToDB(volumeNameA, omMetadataManager); +TestOMRequestUtils.addVolumeToDB(volumeNameB, omMetadataManager); +addBucketsToCache(volumeNameA, ozoneBucket); +addBucketsToCache(volumeNameB, hadoopBucket); + + +String prefixKeyA = "key-a"; +String prefixKeyB = "key-b"; +TreeSet keysASet = new TreeSet<>(); +TreeSet keysBSet = new TreeSet<>(); +for (int i=1; i<= 100; i++) { + if (i % 2 == 0) { +keysASet.add( +prefixKeyA + i); +addKeysToOM(volumeNameA, ozoneBucket, prefixKeyA + i, i); + } else { +keysBSet.add( +prefixKeyB + i); +addKeysToOM(volumeNameA, hadoopBucket, prefixKeyB + i, i); + } +} + + +TreeSet keysAVolumeBSet = new TreeSet<>(); +TreeSet keysBVolumeBSet = new TreeSet<>(); +for (int i=1; i<= 100; i++) { + if (i % 2 == 0) { +keysAVolumeBSet.add( +prefixKeyA + i); +addKeysToOM(volumeNameB, ozoneBucket, prefixKeyA + i, i); + } else { +keysBVolumeBSet.add( +prefixKeyB + i); +addKeysToOM(volumeNameB, hadoopBucket, prefixKeyB + i, i); + } +} + + +// List all keys which have prefix "key-a" +List omKeyInfoList = +omMetadataManager.listKeys(volumeNameA, ozoneBucket, +null, prefixKeyA, 100); + +Assert.assertEquals(omKeyInfoList.size(), 50); + +for (OmKeyInfo omKeyInfo : omKeyInfoList) { + Assert.assertTrue(omKeyInfo.getKeyName().startsWith( + prefixKeyA)); +} + + +String startKey = prefixKeyA + 10; +omKeyInfoList = +omMetadataManager.listKeys(volumeNameA, ozoneBucket, +startKey, prefixKeyA, 100); + +Assert.assertEquals(keysASet.tailSet( +startKey).size() - 1, omKeyInfoList.size()); + +startKey = prefixKeyA + 38; +omKeyInfoList = +omMetadataManager.listKeys(volumeNameA, ozoneBucket, +startKey, prefixKeyA, 100); + +Assert.assertEquals(keysASet.tailSet( +startKey).size() - 1, omKeyInfoList.size()); + +for (OmKeyInfo omKeyInfo : omKeyInfoList) { + Assert.assertTrue(omKeyInfo.getKeyName().startsWith( + prefixKeyA)); + Assert.assertFalse(omKeyInfo.getBucketName().equals( +
[GitHub] [hadoop] arp7 commented on a change in pull request #1588: HDDS-1986. Fix listkeys API.
arp7 commented on a change in pull request #1588: HDDS-1986. Fix listkeys API. URL: https://github.com/apache/hadoop/pull/1588#discussion_r332303770 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -680,26 +688,85 @@ public boolean isBucketEmpty(String volume, String bucket) seekPrefix = getBucketKey(volumeName, bucketName + OM_KEY_PREFIX); } int currentCount = 0; -try (TableIterator> keyIter = -getKeyTable() -.iterator()) { - KeyValue kv = keyIter.seek(seekKey); - while (currentCount < maxKeys && keyIter.hasNext()) { -kv = keyIter.next(); -// Skip the Start key if needed. -if (kv != null && skipStartKey && kv.getKey().equals(seekKey)) { - continue; + + +TreeMap cacheKeyMap = new TreeMap<>(); +Set deletedKeySet = new TreeSet<>(); +Iterator, CacheValue>> iterator = +keyTable.cacheIterator(); + +//TODO: We can avoid this iteration if table cache has stored entries in +// treemap. Currently HashMap is used in Cache. HashMap get operation is an +// constant time operation, where as for treeMap get is log(n). +// So if we move to treemap, the get operation will be affected. As get +// is frequent operation on table. So, for now in list we iterate cache map +// and construct treeMap which match with keyPrefix and are greater than or +// equal to startKey. Later we can revisit this, if list operation +// is becoming slow. +while (iterator.hasNext()) { + Map.Entry< CacheKey, CacheValue> entry = + iterator.next(); + + String key = entry.getKey().getCacheKey(); + OmKeyInfo omKeyInfo = entry.getValue().getCacheValue(); + // Making sure that entry in cache is not for delete key request. + + if (omKeyInfo != null) { +if (key.startsWith(seekPrefix) && key.compareTo(seekKey) >= 0) { + cacheKeyMap.put(key, omKeyInfo); } + } else { +deletedKeySet.add(key); + } +} + +// Get maxKeys from DB if it has. + +try (TableIterator> + keyIter = getKeyTable().iterator()) { + KeyValue< String, OmKeyInfo > kv; + keyIter.seek(seekKey); + // we need to iterate maxKeys + 1 here because if skipStartKey is true, + // we should skip that entry and return the result. + while (currentCount < maxKeys + 1 && keyIter.hasNext()) { +kv = keyIter.next(); if (kv != null && kv.getKey().startsWith(seekPrefix)) { - result.add(kv.getValue()); - currentCount++; + + // Entry should not be marked for delete, consider only those + // entries. + if(!deletedKeySet.contains(kv.getKey())) { +cacheKeyMap.put(kv.getKey(), kv.getValue()); +currentCount++; + } } else { // The SeekPrefix does not match any more, we can break out of the // loop. break; } } } + +// Finally DB entries and cache entries are merged, then return the count +// of maxKeys from the sorted map. +currentCount = 0; + +for (Map.Entry cacheKey : cacheKeyMap.entrySet()) { Review comment: The second iteration is unfortunate. We should see if there is a way to avoid it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946364#comment-16946364 ] Hadoop QA commented on HADOOP-16152: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 51s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 28s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 7s{color} | {color:red} root in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 15s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 17s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 32s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m 51s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 51s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 8s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 7s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 29s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 33s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 6s{color} | {color:green} hadoop-yarn-applications-catalog-webapp in the patch passed. {color} | | {colo
[GitHub] [hadoop] hadoop-yetus commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations
hadoop-yetus commented on issue #1582: HDDS-2217. Removed redundant LOG4J lines from docker configurations URL: https://github.com/apache/hadoop/pull/1582#issuecomment-539264930 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 11 | https://github.com/apache/hadoop/pull/1582 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1582 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jeffsaremi commented on a change in pull request #1480: HDFS-14857. FS operations fail in HA mode: DataNode fails to connect to NameNode
jeffsaremi commented on a change in pull request #1480: HDFS-14857. FS operations fail in HA mode: DataNode fails to connect to NameNode URL: https://github.com/apache/hadoop/pull/1480#discussion_r332286495 ## File path: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java ## @@ -61,7 +63,10 @@ public ConfiguredFailoverProxyProvider(Configuration conf, URI uri, } @Override - public void performFailover(T currentProxy) { + public void performFailover(T currentProxy) { +//reset the IP address in case the stale IP was the cause for failover +LOG.info("Resetting cached proxy: " + currentProxyIndex); +resetProxyAddress(proxies, currentProxyIndex); Review comment: We don't know if the next proxy will have the same issue or not. We just know about the current proxy having issue so we can't just alter the next one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
bharatviswa504 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332284284 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String bucket) } int currentCount = 0; -try (TableIterator> -bucketIter = bucketTable.iterator()) { - KeyValue kv = bucketIter.seek(startKey); - while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) { -kv = bucketIter.next(); -// Skip the Start Bucket if needed. -if (kv != null && skipStartKey && -kv.getKey().equals(startKey)) { + +// For Bucket it is full cache, so we can just iterate in-memory table +// cache. +Iterator, CacheValue>> iterator = Review comment: To improve this API, I think we should change our cache data structure which will be effective for both read/write/list API. (Initially, usage of ConcurrentHashMap, as get() is a constant time operation, but it has caused the problem for list API). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
bharatviswa504 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332282739 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String bucket) } int currentCount = 0; -try (TableIterator> -bucketIter = bucketTable.iterator()) { - KeyValue kv = bucketIter.seek(startKey); - while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) { -kv = bucketIter.next(); -// Skip the Start Bucket if needed. -if (kv != null && skipStartKey && -kv.getKey().equals(startKey)) { + +// For Bucket it is full cache, so we can just iterate in-memory table +// cache. +Iterator, CacheValue>> iterator = Review comment: There is a maxCount, when we reach the count, we will return immediately, in that case, we shall not iterate all volumes in a bucket. BucketTable cache is just a concurrentHashMap of all buckets in OM. So let's take an example: As it is ConcurrentSkipListMap, it has items sorted based on the key. We have entries in bucket table cache like below: (For brevity, removed bucketInfo structures which are the values for the key.) /vol/buck1 /vol/buck2 /vol/buck3 /vol2/bucket2 /vol2/bucket3 /vol2/bucket4 When we want to list buckets of /vol, and in each return only 1 entry(maximum count), we return /vol/buck1 and then immediately return, next time listBuckets is called, startkey will be /vol/buck1,1, we return /vol/buck2, (To return this we iterate 2 entries), like that so on. But when we want to list all buckets is /vol2, we will iterate the entries from start, and reach to /vol2 in the cache and once the maximum count is reached we return from there. So, to answer your question not every time we iterate entire cache map. (But some times we iterate and skip them, as shown with case of vol2 bucket list) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
bharatviswa504 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332282739 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String bucket) } int currentCount = 0; -try (TableIterator> -bucketIter = bucketTable.iterator()) { - KeyValue kv = bucketIter.seek(startKey); - while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) { -kv = bucketIter.next(); -// Skip the Start Bucket if needed. -if (kv != null && skipStartKey && -kv.getKey().equals(startKey)) { + +// For Bucket it is full cache, so we can just iterate in-memory table +// cache. +Iterator, CacheValue>> iterator = Review comment: There is a maxCount, when we reach the count, we will return immediately, in that case, we shall not iterate all volumes in a bucket. BucketTable cache is just a concurrentHashMap of all buckets in OM. So let's take an example: As it is ConcurrentSkipListMap, it has items sorted based on the key. We have entries in bucket table cache like below: (For brevity, removed bucketInfo structures which are the values for the key.) /vol/buck1 /vol/buck2 /vol/buck3 /vol2/bucket2 /vol2/bucket3 /vol2/bucket4 When we want to list buckets of /vol, and in each return only 1 entry(maximum count), we return /vol/buck1 and then immediately return, next time listBuckets is called, startkey will be /vol/buck1,1, we return /vol/buck2, (To return this we iterate 2 entries), like that so on. But when we want to list all buckets is /vol2, we will iterate the entries from start, and reach to /vol2 in the cache and once the maximum count is reached we return from there. So, to answer your question not every time we iterate entire cache map. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
bharatviswa504 commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332281050 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java ## @@ -53,4 +53,18 @@ public boolean equals(Object o) { public int hashCode() { return Objects.hash(key); } + + @Override + public int compareTo(Object o) { +if(Objects.equals(key, ((CacheKey)o).key)) { + return 0; +} else { + if (key instanceof String) { +return ((String) key).compareTo((String) ((CacheKey)o).key); + } else { +// If not type string, convert to string and compare. +return key.toString().compareToCacheKey) o).key).toString()); Review comment: When CacheKey KEY type is not string. For BucketTable type is string, for tokentable type is OzoneTokenIdentifier. As this is common class used by all tables in OM, added the if, else condition. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946329#comment-16946329 ] Íñigo Goiri commented on HADOOP-16579: -- Thanks [~weichiu] for the ping. I'll try to deploy trunk to see if there's something broken. We internally use Hadoop 2.9 with ZooKeeper 3.4.8 and there were not many issues when switching from 3.4.6. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer edited a comment on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
anuengineer edited a comment on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#issuecomment-539237587 > Right now ActiveLock creates ReadWrite Lock with non-fair mode. Do you mean, we want to create the RWLOCK with an option of fair mode. If my understanding is wrong, could you let me know what additional things need to be implemented? When you use a Reader writer lock, there is a question of fairness. Where as exclusive locks are first come first serve. > > And also this work is mainly to improve read performance workloads, as now with current approach of exclusive lock all reads are serialized. I am afraid this gives so much importance to Reads that you will have your writes getting stalled completely. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
anuengineer commented on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#issuecomment-539237587 > Right now ActiveLock creates ReadWrite Lock with non-fair mode. Do you mean, we want to create the RWLOCK with an option of fair mode. If my understanding is wrong, could you let me know what additional things need to be implemented? When you use a Reader writer lock, there is a question of fairness. Where as exclusive locks are first come first serve. > > And also this work is mainly to improve read performance workloads, as now with current approach of exclusive lock all reads are serialized. I am afraid this gives so much importance to Reads that you will have your writes getting stalled completely. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
anuengineer commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#discussion_r332270946 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java ## @@ -105,15 +109,66 @@ public OzoneManagerLock(Configuration conf) { * should be bucket name. For remaining all resource only one param should * be passed. */ + @Deprecated public boolean acquireLock(Resource resource, String... resources) { String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + /** + * Acquire read lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread + * acquiring lock again is not allowed. + * + * Special Note for USER_LOCK: Single thread can acquire single user lock/ + * multi user lock. But not both at the same time. + * @param resource - Type of the resource. + * @param resources - Resource names on which user want to acquire lock. + * For Resource type BUCKET_LOCK, first param should be volume, second param + * should be bucket name. For remaining all resource only one param should + * be passed. + */ + public boolean acquireReadLock(Resource resource, String... resources) { +String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::readLock, READ_LOCK); + } + + + /** + * Acquire write lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread + * acquiring lock again is not allowed. + * + * Special Note for USER_LOCK: Single thread can acquire single user lock/ + * multi user lock. But not both at the same time. + * @param resource - Type of the resource. + * @param resources - Resource names on which user want to acquire lock. + * For Resource type BUCKET_LOCK, first param should be volume, second param + * should be bucket name. For remaining all resource only one param should + * be passed. + */ + public boolean acquireWriteLock(Resource resource, String... resources) { +String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + private boolean lock(Resource resource, String resourceName, + Consumer lockFn, String lockType) { if (!resource.canLock(lockSet.get())) { String errorMessage = getErrorMessage(resource); LOG.error(errorMessage); throw new RuntimeException(errorMessage); } else { - manager.lock(resourceName); - LOG.debug("Acquired {} lock on resource {}", resource.name, + lockFn.accept(resourceName); + LOG.debug("Acquired {} {} lock on resource {}", lockType, resource.name, resourceName); Review comment: Yes, it is very confusing. But Thanks for the explanation, it makes sense now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16638) Use Relative URLs in Hadoop KMS WebApps
[ https://issues.apache.org/jira/browse/HADOOP-16638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946301#comment-16946301 ] Hadoop QA commented on HADOOP-16638: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m 38s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 19s{color} | {color:red} root in trunk failed. {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 43m 44s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 1s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 1s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 27s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 30s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 67m 31s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:1dde3efb91e | | JIRA Issue | HADOOP-16638 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12982422/HADOOP-16638.1.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient | | uname | Linux cd9e6254e8ab 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / cfba6ac | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_222 | | compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/16577/artifact/out/branch-compile-root.txt | | compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/16577/artifact/out/patch-compile-root.txt | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/16577/artifact/out/patch-compile-root.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/16577/testReport/ | | Max. process+thread count | 308 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/16577/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Use Relative URLs in Hadoop KMS WebApps > --- > > Key: HADOOP-16638 > URL: https://issues.apache.org/jira/browse/HADOOP-16638 > Project: Hadoop Common > Issue Type: Sub-task >
[GitHub] [hadoop] anuengineer commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
anuengineer commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332264370 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java ## @@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String bucket) } int currentCount = 0; -try (TableIterator> -bucketIter = bucketTable.iterator()) { - KeyValue kv = bucketIter.seek(startKey); - while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) { -kv = bucketIter.next(); -// Skip the Start Bucket if needed. -if (kv != null && skipStartKey && -kv.getKey().equals(startKey)) { + +// For Bucket it is full cache, so we can just iterate in-memory table +// cache. +Iterator, CacheValue>> iterator = Review comment: Sorry, I am not about to make sure of thi; but for each request do we iterator through the whole bucket space here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1555: HDDS-1984. Fix listBucket API.
anuengineer commented on a change in pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#discussion_r332257684 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java ## @@ -53,4 +53,18 @@ public boolean equals(Object o) { public int hashCode() { return Objects.hash(key); } + + @Override + public int compareTo(Object o) { +if(Objects.equals(key, ((CacheKey)o).key)) { + return 0; +} else { + if (key instanceof String) { +return ((String) key).compareTo((String) ((CacheKey)o).key); + } else { +// If not type string, convert to string and compare. +return key.toString().compareToCacheKey) o).key).toString()); Review comment: when can this happen? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
bharatviswa504 commented on issue #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#issuecomment-539225586 > I have an uber question on this patch. How do we ensure that writes will not be starved on a resource, since Reads allow multiple of them to get thru at the same time? Do we have a mechanism to avoid write starvation in place? if not, it is not better to keep simple locks? Right now ActiveLock creates ReadWrite Lock with non-fair mode. Do you mean, we want to create the RWLOCK with an option of fair mode. If my understanding is wrong, could you let me know what additional things need to be implemented? And also this work is mainly to improve read performance workloads, as now with current approach of exclusive lock all reads are serialized. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
bharatviswa504 commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#discussion_r332255791 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java ## @@ -105,15 +109,66 @@ public OzoneManagerLock(Configuration conf) { * should be bucket name. For remaining all resource only one param should * be passed. */ + @Deprecated public boolean acquireLock(Resource resource, String... resources) { String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + /** + * Acquire read lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread + * acquiring lock again is not allowed. + * + * Special Note for USER_LOCK: Single thread can acquire single user lock/ + * multi user lock. But not both at the same time. + * @param resource - Type of the resource. + * @param resources - Resource names on which user want to acquire lock. + * For Resource type BUCKET_LOCK, first param should be volume, second param + * should be bucket name. For remaining all resource only one param should + * be passed. + */ + public boolean acquireReadLock(Resource resource, String... resources) { +String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::readLock, READ_LOCK); + } + + + /** + * Acquire write lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread + * acquiring lock again is not allowed. + * + * Special Note for USER_LOCK: Single thread can acquire single user lock/ + * multi user lock. But not both at the same time. + * @param resource - Type of the resource. + * @param resources - Resource names on which user want to acquire lock. + * For Resource type BUCKET_LOCK, first param should be volume, second param + * should be bucket name. For remaining all resource only one param should + * be passed. + */ + public boolean acquireWriteLock(Resource resource, String... resources) { +String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + private boolean lock(Resource resource, String resourceName, + Consumer lockFn, String lockType) { if (!resource.canLock(lockSet.get())) { String errorMessage = getErrorMessage(resource); LOG.error(errorMessage); throw new RuntimeException(errorMessage); } else { - manager.lock(resourceName); - LOG.debug("Acquired {} lock on resource {}", resource.name, + lockFn.accept(resourceName); + LOG.debug("Acquired {} {} lock on resource {}", lockType, resource.name, resourceName); Review comment: Here the first resource.name prints VOLUME_LOCK/BUCKET_LOCK, next resourceName prints actual resource name. (I think it is little confusing here, because class Resource name is defined like that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
bharatviswa504 commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#discussion_r332255791 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java ## @@ -105,15 +109,66 @@ public OzoneManagerLock(Configuration conf) { * should be bucket name. For remaining all resource only one param should * be passed. */ + @Deprecated public boolean acquireLock(Resource resource, String... resources) { String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + /** + * Acquire read lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread + * acquiring lock again is not allowed. + * + * Special Note for USER_LOCK: Single thread can acquire single user lock/ + * multi user lock. But not both at the same time. + * @param resource - Type of the resource. + * @param resources - Resource names on which user want to acquire lock. + * For Resource type BUCKET_LOCK, first param should be volume, second param + * should be bucket name. For remaining all resource only one param should + * be passed. + */ + public boolean acquireReadLock(Resource resource, String... resources) { +String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::readLock, READ_LOCK); + } + + + /** + * Acquire write lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread + * acquiring lock again is not allowed. + * + * Special Note for USER_LOCK: Single thread can acquire single user lock/ + * multi user lock. But not both at the same time. + * @param resource - Type of the resource. + * @param resources - Resource names on which user want to acquire lock. + * For Resource type BUCKET_LOCK, first param should be volume, second param + * should be bucket name. For remaining all resource only one param should + * be passed. + */ + public boolean acquireWriteLock(Resource resource, String... resources) { +String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + private boolean lock(Resource resource, String resourceName, + Consumer lockFn, String lockType) { if (!resource.canLock(lockSet.get())) { String errorMessage = getErrorMessage(resource); LOG.error(errorMessage); throw new RuntimeException(errorMessage); } else { - manager.lock(resourceName); - LOG.debug("Acquired {} lock on resource {}", resource.name, + lockFn.accept(resourceName); + LOG.debug("Acquired {} {} lock on resource {}", lockType, resource.name, resourceName); Review comment: Here the first resource.name prints VOLUME_LOCK/BUCKET_LOCK, next resourceName prints actual resource name. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager.
anuengineer commented on a change in pull request #1589: HDDS-2244. Use new ReadWrite lock in OzoneManager. URL: https://github.com/apache/hadoop/pull/1589#discussion_r332251450 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java ## @@ -105,15 +109,66 @@ public OzoneManagerLock(Configuration conf) { * should be bucket name. For remaining all resource only one param should * be passed. */ + @Deprecated public boolean acquireLock(Resource resource, String... resources) { String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + /** + * Acquire read lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread + * acquiring lock again is not allowed. + * + * Special Note for USER_LOCK: Single thread can acquire single user lock/ + * multi user lock. But not both at the same time. + * @param resource - Type of the resource. + * @param resources - Resource names on which user want to acquire lock. + * For Resource type BUCKET_LOCK, first param should be volume, second param + * should be bucket name. For remaining all resource only one param should + * be passed. + */ + public boolean acquireReadLock(Resource resource, String... resources) { +String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::readLock, READ_LOCK); + } + + + /** + * Acquire write lock on resource. + * + * For S3_BUCKET_LOCK, VOLUME_LOCK, BUCKET_LOCK type resource, same + * thread acquiring lock again is allowed. + * + * For USER_LOCK, PREFIX_LOCK, S3_SECRET_LOCK type resource, same thread + * acquiring lock again is not allowed. + * + * Special Note for USER_LOCK: Single thread can acquire single user lock/ + * multi user lock. But not both at the same time. + * @param resource - Type of the resource. + * @param resources - Resource names on which user want to acquire lock. + * For Resource type BUCKET_LOCK, first param should be volume, second param + * should be bucket name. For remaining all resource only one param should + * be passed. + */ + public boolean acquireWriteLock(Resource resource, String... resources) { +String resourceName = generateResourceName(resource, resources); +return lock(resource, resourceName, manager::writeLock, WRITE_LOCK); + } + + private boolean lock(Resource resource, String resourceName, + Consumer lockFn, String lockType) { if (!resource.canLock(lockSet.get())) { String errorMessage = getErrorMessage(resource); LOG.error(errorMessage); throw new RuntimeException(errorMessage); } else { - manager.lock(resourceName); - LOG.debug("Acquired {} lock on resource {}", resource.name, + lockFn.accept(resourceName); + LOG.debug("Acquired {} {} lock on resource {}", lockType, resource.name, resourceName); Review comment: I am trying to read this debug statement.. Do you need to have resource name twice ? once via resource.name and another via resourceName? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1559: HDDS-1737. Add Volume check in KeyManager and File Operations.
bharatviswa504 commented on a change in pull request #1559: HDDS-1737. Add Volume check in KeyManager and File Operations. URL: https://github.com/apache/hadoop/pull/1559#discussion_r332253834 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java ## @@ -123,10 +126,17 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, acquiredLock = omMetadataManager.getLock().acquireLock(BUCKET_LOCK, volumeName, bucketName); - // Not doing bucket/volume checks here. In this way we can avoid db - // checks for them. - // TODO: Once we have volume/bucket full cache, we can add - // them back, as these checks will be inexpensive at that time. + // Check volume exist. + if (omMetadataManager.getVolumeTable().isExist(volumeName)) { +throw new OMException("Volume not found " + volumeName, Review comment: Same as above This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1559: HDDS-1737. Add Volume check in KeyManager and File Operations.
bharatviswa504 commented on a change in pull request #1559: HDDS-1737. Add Volume check in KeyManager and File Operations. URL: https://github.com/apache/hadoop/pull/1559#discussion_r332252751 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java ## @@ -117,12 +121,19 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, acquiredLock = omMetadataManager.getLock().acquireLock(BUCKET_LOCK, volumeName, bucketName); - // Not doing bucket/volume checks here. In this way we can avoid db - // checks for them. - // TODO: Once we have volume/bucket full cache, we can add - // them back, as these checks will be inexpensive at that time. - OmKeyInfo omKeyInfo = omMetadataManager.getKeyTable().get(objectKey); + // Check volume exist. + if (omMetadataManager.getVolumeTable().isExist(volumeName)) { Review comment: Here it should be if (!omMetadataManager.getVolumeTable().isExist(volumeName)) right? And also we should pass omMetadataManagerImpl.getVolumeKey/getBucketKey, not direct volumeName/bucketName. As here if it does not exist, we should return error? ``` /** * Check if a given key exists in Metadata store. * (Optimization to save on data deserialization) * A lock on the key / bucket needs to be acquired before invoking this API. * @param key metadata key * @return true if the metadata store contains a key. * @throws IOException on Failure */ boolean isExist(KEY key) throws IOException; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1559: HDDS-1737. Add Volume check in KeyManager and File Operations.
bharatviswa504 commented on a change in pull request #1559: HDDS-1737. Add Volume check in KeyManager and File Operations. URL: https://github.com/apache/hadoop/pull/1559#discussion_r332253241 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java ## @@ -117,12 +121,19 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, acquiredLock = omMetadataManager.getLock().acquireLock(BUCKET_LOCK, volumeName, bucketName); - // Not doing bucket/volume checks here. In this way we can avoid db - // checks for them. - // TODO: Once we have volume/bucket full cache, we can add - // them back, as these checks will be inexpensive at that time. - OmKeyInfo omKeyInfo = omMetadataManager.getKeyTable().get(objectKey); + // Check volume exist. + if (omMetadataManager.getVolumeTable().isExist(volumeName)) { Review comment: And also we can do little optimization here, check the first bucket exists or not, if it does not exist, then check volume? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1559: HDDS-1737. Add Volume check in KeyManager and File Operations.
bharatviswa504 commented on a change in pull request #1559: HDDS-1737. Add Volume check in KeyManager and File Operations. URL: https://github.com/apache/hadoop/pull/1559#discussion_r332252751 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java ## @@ -117,12 +121,19 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, acquiredLock = omMetadataManager.getLock().acquireLock(BUCKET_LOCK, volumeName, bucketName); - // Not doing bucket/volume checks here. In this way we can avoid db - // checks for them. - // TODO: Once we have volume/bucket full cache, we can add - // them back, as these checks will be inexpensive at that time. - OmKeyInfo omKeyInfo = omMetadataManager.getKeyTable().get(objectKey); + // Check volume exist. + if (omMetadataManager.getVolumeTable().isExist(volumeName)) { Review comment: Here it should be if (!omMetadataManager.getVolumeTable().isExist(volumeName)) right? As here if it does not exist, we should return error? ``` /** * Check if a given key exists in Metadata store. * (Optimization to save on data deserialization) * A lock on the key / bucket needs to be acquired before invoking this API. * @param key metadata key * @return true if the metadata store contains a key. * @throws IOException on Failure */ boolean isExist(KEY key) throws IOException; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on issue #1610: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete.
swagle commented on issue #1610: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete. URL: https://github.com/apache/hadoop/pull/1610#issuecomment-539217658 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer closed pull request #1606: HDDS-2262. SLEEP_SECONDS: command not found
anuengineer closed pull request #1606: HDDS-2262. SLEEP_SECONDS: command not found URL: https://github.com/apache/hadoop/pull/1606 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1606: HDDS-2262. SLEEP_SECONDS: command not found
anuengineer commented on issue #1606: HDDS-2262. SLEEP_SECONDS: command not found URL: https://github.com/apache/hadoop/pull/1606#issuecomment-539216038 Thank you for the contribution. I have committed this patch to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer closed pull request #1605: HDDS-2259. Container Data Scrubber computes wrong checksum
anuengineer closed pull request #1605: HDDS-2259. Container Data Scrubber computes wrong checksum URL: https://github.com/apache/hadoop/pull/1605 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1605: HDDS-2259. Container Data Scrubber computes wrong checksum
anuengineer commented on issue #1605: HDDS-2259. Container Data Scrubber computes wrong checksum URL: https://github.com/apache/hadoop/pull/1605#issuecomment-539214958 +1. LGTM. Thank you for fixing this very important issue. I have committed this patch to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16639) Use Relative URLS in Hadoop HDFS HTTP FS
[ https://issues.apache.org/jira/browse/HADOOP-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor resolved HADOOP-16639. - Resolution: Won't Fix Belong in HDFS project > Use Relative URLS in Hadoop HDFS HTTP FS > > > Key: HADOOP-16639 > URL: https://issues.apache.org/jira/browse/HADOOP-16639 > Project: Hadoop Common > Issue Type: Sub-task > Components: httpfs >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16639) Use Relative URLS in Hadoop HDFS HTTP FS
[ https://issues.apache.org/jira/browse/HADOOP-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946250#comment-16946250 ] David Mollitor edited comment on HADOOP-16639 at 10/7/19 9:20 PM: -- Belongs in HDFS project was (Author: belugabehr): Belong in HDFS project > Use Relative URLS in Hadoop HDFS HTTP FS > > > Key: HADOOP-16639 > URL: https://issues.apache.org/jira/browse/HADOOP-16639 > Project: Hadoop Common > Issue Type: Sub-task > Components: httpfs >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15956) Use relative resource URLs across WebUI components
[ https://issues.apache.org/jira/browse/HADOOP-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HADOOP-15956: Status: Open (was: Patch Available) > Use relative resource URLs across WebUI components > -- > > Key: HADOOP-15956 > URL: https://issues.apache.org/jira/browse/HADOOP-15956 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Greg Phillips >Assignee: Greg Phillips >Priority: Minor > Attachments: HADOOP-15956.001.patch > > > Similar to HDFS-12961 there are absolute paths used for static resources in > the WebUI for HDFS & KMS which can cause issues when attempting to access > these pages via a reverse proxy. Using relative paths in all WebUI components > will allow pages to render properly when using a reverse proxy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer closed pull request #1607: HDDS-2264. Improve output of TestOzoneContainer
anuengineer closed pull request #1607: HDDS-2264. Improve output of TestOzoneContainer URL: https://github.com/apache/hadoop/pull/1607 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1607: HDDS-2264. Improve output of TestOzoneContainer
anuengineer commented on issue #1607: HDDS-2264. Improve output of TestOzoneContainer URL: https://github.com/apache/hadoop/pull/1607#issuecomment-539207650 +1. LGTM. I have committed this patch to the trunk. Thank you for the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16639) Use Relative URLS in Hadoop HDFS HTTP FS
David Mollitor created HADOOP-16639: --- Summary: Use Relative URLS in Hadoop HDFS HTTP FS Key: HADOOP-16639 URL: https://issues.apache.org/jira/browse/HADOOP-16639 Project: Hadoop Common Issue Type: Sub-task Components: httpfs Affects Versions: 3.2.0 Reporter: David Mollitor Assignee: David Mollitor -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16638) Use Relative URLs in Hadoop KMS WebApps
[ https://issues.apache.org/jira/browse/HADOOP-16638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HADOOP-16638: Status: Patch Available (was: Open) > Use Relative URLs in Hadoop KMS WebApps > --- > > Key: HADOOP-16638 > URL: https://issues.apache.org/jira/browse/HADOOP-16638 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HADOOP-16638.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16638) Use Relative URLs in Hadoop KMS WebApps
[ https://issues.apache.org/jira/browse/HADOOP-16638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HADOOP-16638: Flags: Patch > Use Relative URLs in Hadoop KMS WebApps > --- > > Key: HADOOP-16638 > URL: https://issues.apache.org/jira/browse/HADOOP-16638 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HADOOP-16638.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16638) Use Relative URLs in Hadoop KMS WebApps
[ https://issues.apache.org/jira/browse/HADOOP-16638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HADOOP-16638: Attachment: HADOOP-16638.1.patch > Use Relative URLs in Hadoop KMS WebApps > --- > > Key: HADOOP-16638 > URL: https://issues.apache.org/jira/browse/HADOOP-16638 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HADOOP-16638.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
anuengineer commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590#issuecomment-539204102 Thank you for the contribution. I have committed this change to the trunk branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer closed pull request #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
anuengineer closed pull request #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
anuengineer commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590#issuecomment-539203108 +1. LGTM. There is one thing that is not very clear to me. Why add container it is not an issue, but I am not sure I understand the benefit either. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16638) Use Relative URLs in Hadoop KMS WebApps
[ https://issues.apache.org/jira/browse/HADOOP-16638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HADOOP-16638: Summary: Use Relative URLs in Hadoop KMS WebApps (was: Use Relative URLs in Hadoop KMS) > Use Relative URLs in Hadoop KMS WebApps > --- > > Key: HADOOP-16638 > URL: https://issues.apache.org/jira/browse/HADOOP-16638 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16638) Use Relative URLs in Hadoop KMS
David Mollitor created HADOOP-16638: --- Summary: Use Relative URLs in Hadoop KMS Key: HADOOP-16638 URL: https://issues.apache.org/jira/browse/HADOOP-16638 Project: Hadoop Common Issue Type: Sub-task Components: kms Affects Versions: 3.2.0 Reporter: David Mollitor Assignee: David Mollitor -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16152: Attachment: HADOOP-16152.006.patch Status: Patch Available (was: In Progress) > Upgrade Eclipse Jetty version to 9.4.x > -- > > Key: HADOOP-16152 > URL: https://issues.apache.org/jira/browse/HADOOP-16152 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Yuming Wang >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, > HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.005.patch, > HADOOP-16152.006.patch, HADOOP-16152.v1.patch > > > Some big data projects have been upgraded Jetty to 9.4.x, which causes some > compatibility issues. > Spark: > [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146] > Calcite: > [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87] > Hive: HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16152: Status: In Progress (was: Patch Available) > Upgrade Eclipse Jetty version to 9.4.x > -- > > Key: HADOOP-16152 > URL: https://issues.apache.org/jira/browse/HADOOP-16152 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Yuming Wang >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, > HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.005.patch, > HADOOP-16152.006.patch, HADOOP-16152.v1.patch > > > Some big data projects have been upgraded Jetty to 9.4.x, which causes some > compatibility issues. > Spark: > [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146] > Calcite: > [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87] > Hive: HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946216#comment-16946216 ] Siyao Meng commented on HADOOP-16152: - rev 006 changelog: addressing checkstyle, whitespace and findbugs. Apart from ZK unit test failures, *TestRenameWithSnapshots#testRename2PreDescendant* is also unrelated (failed even before this patch, some commit else probably broke it). > Upgrade Eclipse Jetty version to 9.4.x > -- > > Key: HADOOP-16152 > URL: https://issues.apache.org/jira/browse/HADOOP-16152 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Yuming Wang >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, > HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.005.patch, > HADOOP-16152.v1.patch > > > Some big data projects have been upgraded Jetty to 9.4.x, which causes some > compatibility issues. > Spark: > [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146] > Calcite: > [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87] > Hive: HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1611: Hadoop 16612 Track Azure Blob File System client-perceived latency
hadoop-yetus commented on issue #1611: Hadoop 16612 Track Azure Blob File System client-perceived latency URL: https://github.com/apache/hadoop/pull/1611#issuecomment-539195753 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1087 | trunk passed | | +1 | compile | 33 | trunk passed | | +1 | checkstyle | 26 | trunk passed | | +1 | mvnsite | 36 | trunk passed | | +1 | shadedclient | 793 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 27 | trunk passed | | 0 | spotbugs | 52 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 51 | trunk passed | | -0 | patch | 79 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 29 | the patch passed | | +1 | compile | 25 | the patch passed | | +1 | javac | 25 | the patch passed | | -0 | checkstyle | 20 | hadoop-tools/hadoop-azure: The patch generated 71 new + 5 unchanged - 0 fixed = 76 total (was 5) | | +1 | mvnsite | 28 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 777 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 25 | the patch passed | | +1 | findbugs | 56 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 84 | hadoop-azure in the patch passed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 3283 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1611/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1611 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 8da536829358 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 382967b | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1611/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1611/1/testReport/ | | Max. process+thread count | 411 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1611/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on issue #1610: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete.
swagle commented on issue #1610: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete. URL: https://github.com/apache/hadoop/pull/1610#issuecomment-539184469 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster
adoroszlai commented on issue #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590#issuecomment-539179385 @anuengineer please review, too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jeeteshm opened a new pull request #1611: Hadoop 16612 Track Azure Blob File System client-perceived latency
jeeteshm opened a new pull request #1611: Hadoop 16612 Track Azure Blob File System client-perceived latency URL: https://github.com/apache/hadoop/pull/1611 Add instrumentation code to measure the ADLS Gen 2 API performance Add a feature switch to optionally enable this feature Add unit tests for correctness and performance This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jeeteshm commented on issue #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency
jeeteshm commented on issue #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency URL: https://github.com/apache/hadoop/pull/1569#issuecomment-539169875 Raising a new PR because this one shows changes that don't belong to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jeeteshm opened a new pull request #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency
jeeteshm opened a new pull request #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency URL: https://github.com/apache/hadoop/pull/1569 Add instrumentation code to measure the ADLS Gen 2 API performance Add a feature switch to optionally enable this feature Add unit tests for correctness and performance This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jeeteshm closed pull request #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency
jeeteshm closed pull request #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency URL: https://github.com/apache/hadoop/pull/1569 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jeeteshm closed pull request #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency
jeeteshm closed pull request #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency URL: https://github.com/apache/hadoop/pull/1569 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1610: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete.
hadoop-yetus commented on issue #1610: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete. URL: https://github.com/apache/hadoop/pull/1610#issuecomment-539167261 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 45 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 26 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 22 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 843 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 19 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 945 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 33 | hadoop-hdds in trunk failed. | | -1 | findbugs | 20 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 31 | Maven dependency ordering for patch | | -1 | mvninstall | 48 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 38 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 19 | hadoop-ozone in the patch failed. | | -1 | cc | 25 | hadoop-hdds in the patch failed. | | -1 | cc | 19 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 19 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 27 | hadoop-hdds: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) | | -0 | checkstyle | 30 | hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 723 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in the patch failed. | | -1 | javadoc | 21 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 27 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2438 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1610 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc xml | | uname | Linux 5d4b009da9f3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9685a6c | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1610/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | c
[GitHub] [hadoop] hadoop-yetus commented on issue #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency
hadoop-yetus commented on issue #1569: HADOOP-16612 Track Azure Blob File System client-perceived latency URL: https://github.com/apache/hadoop/pull/1569#issuecomment-539166965 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 12 | https://github.com/apache/hadoop/pull/1569 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1569 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1569/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1608: HDDS-2265. integration.sh may report false negative
anuengineer commented on a change in pull request #1608: HDDS-2265. integration.sh may report false negative URL: https://github.com/apache/hadoop/pull/1608#discussion_r332190504 ## File path: hadoop-ozone/dev-support/checks/_mvn_unit_report.sh ## @@ -45,6 +45,11 @@ grep -A1 'Crashed tests' "${REPORT_DIR}/output.log" \ | cut -f2- -d' ' \ | sort -u >> "${REPORT_DIR}/summary.txt" +## Check if Maven was killed +if grep -q 'Killed.* mvn .* test ' "${REPORT_DIR}/output.log"; then Review comment: So we are presuming that Killed will never be used by a test? :) I am fine with that. +1. LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946154#comment-16946154 ] Wei-Chiu Chuang commented on HADOOP-16579: -- Thank you, Norbert. Our unit test doesn't actually run any useful tests when you update only the pom.xml file. We probably want to update to zookeeper 3.5.6, assuming it doesn't have performance regression (we have heavy dependency on zookeeper for KMS and Router Based Federation), and backward compat. We can also gradually get rid of netty3. [~inigoiri] FYI, like I mentioned, RBF have heavy dependency on ZK, so you should care about it. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1586: HDDS-2240. Command line tool for OM HA.
anuengineer commented on a change in pull request #1586: HDDS-2240. Command line tool for OM HA. URL: https://github.com/apache/hadoop/pull/1586#discussion_r332187600 ## File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto ## @@ -1097,11 +1097,34 @@ message UpdateGetS3SecretRequest { required string awsSecret = 2; } +message OMServiceId { +required string serviceID = 1; +} + +/** + This proto is used to define the OM node Id and its ratis server state. +*/ +message RoleInfo { +required string omNodeID = 1; +required string ratisServerRole = 2; +} + +/** + This is used to get the Server States of OMs. +*/ +message ServiceState { +repeated RoleInfo roleInfos = 1; +} + /** The OM service that takes care of Ozone namespace. */ service OzoneManagerService { // A client-to-OM RPC to send client requests to OM Ratis server rpc submitRequest(OMRequest) returns(OMResponse); + +// A client-to-OM RPC to get ratis server states of OMs Review comment: Also just wondering, is there a reason to add a new RPC and not use the submitRequet pattern -- that is add the message to the OMRequest / OMResponse pattern? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1586: HDDS-2240. Command line tool for OM HA.
anuengineer commented on a change in pull request #1586: HDDS-2240. Command line tool for OM HA. URL: https://github.com/apache/hadoop/pull/1586#discussion_r332186630 ## File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto ## @@ -1097,11 +1097,34 @@ message UpdateGetS3SecretRequest { required string awsSecret = 2; } +message OMServiceId { +required string serviceID = 1; +} + +/** + This proto is used to define the OM node Id and its ratis server state. +*/ +message RoleInfo { +required string omNodeID = 1; +required string ratisServerRole = 2; +} + +/** + This is used to get the Server States of OMs. +*/ +message ServiceState { +repeated RoleInfo roleInfos = 1; +} + /** The OM service that takes care of Ozone namespace. */ service OzoneManagerService { // A client-to-OM RPC to send client requests to OM Ratis server rpc submitRequest(OMRequest) returns(OMResponse); + +// A client-to-OM RPC to get ratis server states of OMs +rpc getServiceState(OMServiceId) + returns(ServiceState); } Review comment: What is the use case for this RPC? it is used only from the OM HA tool? or is some thing that clients need to know when communicating with the OM? if clients need to know this info, then clients already make a call called getServiceList and this info is more appropriate to be part of that call? Then both clients and tools like OM HA can get to that information pretty easily. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16637) Fix findbugs warnings in hadoop-cos
[ https://issues.apache.org/jira/browse/HADOOP-16637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien reassigned HADOOP-16637: - Assignee: YiSheng Lien > Fix findbugs warnings in hadoop-cos > --- > > Key: HADOOP-16637 > URL: https://issues.apache.org/jira/browse/HADOOP-16637 > Project: Hadoop Common > Issue Type: Bug > Components: fs/cos >Reporter: Akira Ajisaka >Assignee: YiSheng Lien >Priority: Major > > qbt report: > https://lists.apache.org/thread.html/ab1ea4ac6590061cfb2f89183f33f97e92da0e68e67657dbfbda862f@%3Ccommon-dev.hadoop.apache.org%3E > {noformat} >module:hadoop-cloud-storage-project/hadoop-cos >Redundant nullcheck of dir, which is known to be non-null in > org.apache.hadoop.fs.cosn.BufferPool.createDir(String) Redundant null check > at BufferPool.java:is known to be non-null in > org.apache.hadoop.fs.cosn.BufferPool.createDir(String) Redundant null check > at BufferPool.java:[line 66] >org.apache.hadoop.fs.cosn.CosNInputStream$ReadBuffer.getBuffer() may > expose internal representation by returning CosNInputStream$ReadBuffer.buffer > At CosNInputStream.java:by returning CosNInputStream$ReadBuffer.buffer At > CosNInputStream.java:[line 87] >Found reliance on default encoding in > org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFile(String, File, > byte[]):in > org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFile(String, File, > byte[]): new String(byte[]) At CosNativeFileSystemStore.java:[line 199] >Found reliance on default encoding in > org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFileWithRetry(String, > InputStream, byte[], long):in > org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFileWithRetry(String, > InputStream, byte[], long): new String(byte[]) At > CosNativeFileSystemStore.java:[line 178] >org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.uploadPart(File, > String, String, int) may fail to clean up java.io.InputStream Obligation to > clean up resource created at CosNativeFileSystemStore.java:fail to clean up > java.io.InputStream Obligation to clean up resource created at > CosNativeFileSystemStore.java:[line 252] is not discharged > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1601: HADOOP-16635. S3A innerGetFileStatus scans for directories-only still does a HEAD.
hadoop-yetus commented on issue #1601: HADOOP-16635. S3A innerGetFileStatus scans for directories-only still does a HEAD. URL: https://github.com/apache/hadoop/pull/1601#issuecomment-539131837 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1070 | trunk passed | | +1 | compile | 35 | trunk passed | | +1 | checkstyle | 28 | trunk passed | | +1 | mvnsite | 40 | trunk passed | | +1 | shadedclient | 785 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 30 | trunk passed | | 0 | spotbugs | 60 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 59 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 34 | the patch passed | | +1 | compile | 29 | the patch passed | | +1 | javac | 29 | the patch passed | | +1 | checkstyle | 20 | the patch passed | | +1 | mvnsite | 33 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 780 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 27 | the patch passed | | +1 | findbugs | 62 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 90 | hadoop-aws in the patch passed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 3296 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1601/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1601 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 18c4809af70f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1a77a15 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1601/3/testReport/ | | Max. process+thread count | 447 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1601/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle opened a new pull request #1610: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete.
swagle opened a new pull request #1610: HDDS-1868. Ozone pipelines should be marked as ready only after the leader election is complete. URL: https://github.com/apache/hadoop/pull/1610 Ozone pipeline on restart or when first created, start in allocated state. They are moved into open state after all the pipeline have reported to it. However, this potentially can lead into an issue where the pipeline is still not ready to accept any incoming IO operations. The pipelines should be marked as ready only after the leader election is complete and leader is ready to accept incoming IO. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org