[jira] [Commented] (HDFS-13530) NameNode: Fix NullPointerException when getQuotaUsageInt() invoked
[
https://issues.apache.org/jira/browse/HDFS-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465463#comment-16465463
]
genericqa commented on HDFS-13530:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13530 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922208/HDFS-13530.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f7f0f41af893 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e9159db |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-13531) RBF: RouterAdmin supports to set mount table readonly/readwrite
[
https://issues.apache.org/jira/browse/HDFS-13531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465437#comment-16465437
]
genericqa commented on HDFS-13531:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
31s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13531 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922211/HDFS-13531.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6e9c3d60ffa7 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 67f239c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/24143/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24143/testReport/ |
| Max. process+thread count | 962 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24143/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (HDDS-1) Remove SCM Block DB
[ https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465424#comment-16465424 ] Mukul Kumar Singh commented on HDDS-1: -- +1 for committing the latest patch and fixing the rest of the issues in a followup jira. > Remove SCM Block DB > --- > > Key: HDDS-1 > URL: https://issues.apache.org/jira/browse/HDDS-1 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-1.002.patch, HDDS-1.003.patch, HDFS-13504.001.patch > > > The block/key information is maintained by Ozone Master (a.k.a. KSM). This > ticket is opened to remove the redundant block db at SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-19) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-4309324-SNAPSHOT)
[
https://issues.apache.org/jira/browse/HDDS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465423#comment-16465423
]
Mukul Kumar Singh commented on HDDS-19:
---
I tried running Ozone with the latest patch and the datanode is throwing the
following exception.
{code}
06, 2018 11:03:14 PM org.apache.ratis.shaded.io.grpc.netty.NettyServerTransport
notifyTerminated
INFO: Transport failed
java.lang.NoSuchFieldError: CONTEXT_SPAN_KEY
at
org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$ServerTracer.filterContext(CensusTracingModule.java:340)
at
org.apache.ratis.shaded.io.grpc.internal.StatsTraceContext.serverFilterContext(StatsTraceContext.java:121)
at
org.apache.ratis.shaded.io.grpc.internal.ServerImpl$ServerTransportListenerImpl.createContext(ServerImpl.java:482)
at
org.apache.ratis.shaded.io.grpc.internal.ServerImpl$ServerTransportListenerImpl.streamCreated(ServerImpl.java:416)
at
org.apache.ratis.shaded.io.grpc.netty.NettyServerHandler.onHeadersRead(NettyServerHandler.java:455)
at
org.apache.ratis.shaded.io.grpc.netty.NettyServerHandler.access$800(NettyServerHandler.java:98)
at
org.apache.ratis.shaded.io.grpc.netty.NettyServerHandler$FrameListener.onHeadersRead(NettyServerHandler.java:752)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:321)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:605)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:65)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader$1.processFragment(DefaultHttp2FrameReader.java:457)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:464)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:254)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:388)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:254)
at
org.apache.ratis.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:448)
at
org.apache.ratis.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at
org.apache.ratis.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at
org.apache.ratis.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at
org.apache.ratis.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.ratis.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.ratis.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.ratis.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at
org.apache.ratis.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.ratis.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.ratis.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at
org.apache.ratis.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
at
org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at
org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at
org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at
[jira] [Commented] (HDFS-13530) NameNode: Fix NullPointerException when getQuotaUsageInt() invoked
[ https://issues.apache.org/jira/browse/HDFS-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465409#comment-16465409 ] maobaolong commented on HDFS-13530: --- [~liuhongtong] Thank you, this really worth to fix as soon as possible. BTW, you can assign this ticket to yourself. > NameNode: Fix NullPointerException when getQuotaUsageInt() invoked > -- > > Key: HDFS-13530 > URL: https://issues.apache.org/jira/browse/HDFS-13530 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation, namenode >Reporter: liuhongtong >Priority: Major > Attachments: HDFS-13530.001.patch > > > If the directory is nonexistent, getQuotaUsage rpc call will run into > NullPointerException throwed by > FSDirStatAndListingOp.getQuotaUsageInt() . > I think FSDirStatAndListingOp.getQuotaUsageInt() should throw > FileNotFoundException when the directory is nonexistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13531) RBF: RouterAdmin supports to set mount table readonly/readwrite
[ https://issues.apache.org/jira/browse/HDFS-13531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13531: --- Attachment: HDFS-13531.001.patch > RBF: RouterAdmin supports to set mount table readonly/readwrite > --- > > Key: HDFS-13531 > URL: https://issues.apache.org/jira/browse/HDFS-13531 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13531.001.patch > > > RouterAdmin only supports to set mount point read only. > If we want to reset the mount point read write, there is no way. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13531) RBF: RouterAdmin supports to set mount table readonly/readwrite
[ https://issues.apache.org/jira/browse/HDFS-13531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13531: --- Status: Patch Available (was: Open) > RBF: RouterAdmin supports to set mount table readonly/readwrite > --- > > Key: HDFS-13531 > URL: https://issues.apache.org/jira/browse/HDFS-13531 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13531.001.patch > > > RouterAdmin only supports to set mount point read only. > If we want to reset the mount point read write, there is no way. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13531) RBF: RouterAdmin supports to set mount table readonly/readwrite
liuhongtong created HDFS-13531: -- Summary: RBF: RouterAdmin supports to set mount table readonly/readwrite Key: HDFS-13531 URL: https://issues.apache.org/jira/browse/HDFS-13531 Project: Hadoop HDFS Issue Type: Improvement Reporter: liuhongtong RouterAdmin only supports to set mount point read only. If we want to reset the mount point read write, there is no way. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13528) RBF: If a directory exceeds quota limit then quota usage is not refreshed for other mount entries
[
https://issues.apache.org/jira/browse/HDFS-13528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465381#comment-16465381
]
Yiqun Lin edited comment on HDFS-13528 at 5/7/18 3:04 AM:
--
It's a good catch! I can reproduce this case
{noformat}
2018-05-07 10:48:32,075
[org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUpdateService-0]
ERROR router.RouterQuotaUpdateService
(RouterQuotaUpdateService.java:periodicInvoke(115)) - Quota cache updated error.
org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota
(directories and files) is exceeded: quota=3 file count=4
at
org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUsage.verifyNamespaceQuota(RouterQuotaUsage.java:79)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2271)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2233)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getQuotaRemoteLocations(Quota.java:202)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getValidQuotaLocations(Quota.java:113)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getQuotaUsage(Quota.java:94)
at
org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUpdateService.periodicInvoke(RouterQuotaUpdateService.java:88)
at
org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
{noformat}
[~dibyendu_hadoop], I'm +1 for your fix way, but have another two comments:
* We can do the additional try-catch for mount table update behavior in loop,
so that we can avoid potential exception. At least, one mount table updated
error, won't affect others.
* It's would be good to add a unit test in {{TestRouterQuota}}.
was (Author: linyiqun):
It's a good catch! I can reproduce this case
{noformat}
2018-05-07 10:48:32,075
[org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUpdateService-0]
ERROR router.RouterQuotaUpdateService
(RouterQuotaUpdateService.java:periodicInvoke(115)) - Quota cache updated error.
org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota
(directories and files) is exceeded: quota=3 file count=4
at
org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUsage.verifyNamespaceQuota(RouterQuotaUsage.java:79)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2271)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2233)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getQuotaRemoteLocations(Quota.java:202)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getValidQuotaLocations(Quota.java:113)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getQuotaUsage(Quota.java:94)
at
org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUpdateService.periodicInvoke(RouterQuotaUpdateService.java:88)
at
org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
{noformat}
[~dibyendu_hadoop], I'm +1 for your fix way, but have another two comments:
* We can do the additional try-catch for mount table update behavior in loop,
so that we can avoid potential exception.
* It's would be good to add a unit test in {{TestRouterQuota}}.
> RBF: If a directory exceeds quota limit then quota usage is not refreshed for
> other mount entries
> --
>
> Key: HDFS-13528
> URL: https://issues.apache.org/jira/browse/HDFS-13528
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Dibyendu Karmakar
>Priority: Major
>
> If quota limit is exceeded, RouterQuotaUpdateService#periodicInvoke is
> getting QuotaExceededException and it is not updating the quota usage for
> rest of the mount table entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13528) RBF: If a directory exceeds quota limit then quota usage is not refreshed for other mount entries
[
https://issues.apache.org/jira/browse/HDFS-13528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465381#comment-16465381
]
Yiqun Lin commented on HDFS-13528:
--
It's a good catch! I can reproduce this case
{noformat}
2018-05-07 10:48:32,075
[org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUpdateService-0]
ERROR router.RouterQuotaUpdateService
(RouterQuotaUpdateService.java:periodicInvoke(115)) - Quota cache updated error.
org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota
(directories and files) is exceeded: quota=3 file count=4
at
org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUsage.verifyNamespaceQuota(RouterQuotaUsage.java:79)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2271)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2233)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getQuotaRemoteLocations(Quota.java:202)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getValidQuotaLocations(Quota.java:113)
at
org.apache.hadoop.hdfs.server.federation.router.Quota.getQuotaUsage(Quota.java:94)
at
org.apache.hadoop.hdfs.server.federation.router.RouterQuotaUpdateService.periodicInvoke(RouterQuotaUpdateService.java:88)
at
org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
{noformat}
[~dibyendu_hadoop], I'm +1 for your fix way, but have another two comments:
* We can do the additional try-catch for mount table update behavior in loop,
so that we can avoid potential exception.
* It's would be good to add a unit test in {{TestRouterQuota}}.
> RBF: If a directory exceeds quota limit then quota usage is not refreshed for
> other mount entries
> --
>
> Key: HDFS-13528
> URL: https://issues.apache.org/jira/browse/HDFS-13528
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Dibyendu Karmakar
>Priority: Major
>
> If quota limit is exceeded, RouterQuotaUpdateService#periodicInvoke is
> getting QuotaExceededException and it is not updating the quota usage for
> rest of the mount table entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13530) NameNode: Fix NullPointerException when getQuotaUsageInt() invoked
[ https://issues.apache.org/jira/browse/HDFS-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13530: --- Status: Patch Available (was: Open) > NameNode: Fix NullPointerException when getQuotaUsageInt() invoked > -- > > Key: HDFS-13530 > URL: https://issues.apache.org/jira/browse/HDFS-13530 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation, namenode >Reporter: liuhongtong >Priority: Major > Attachments: HDFS-13530.001.patch > > > If the directory is nonexistent, getQuotaUsage rpc call will run into > NullPointerException throwed by > FSDirStatAndListingOp.getQuotaUsageInt() . > I think FSDirStatAndListingOp.getQuotaUsageInt() should throw > FileNotFoundException when the directory is nonexistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13530) NameNode: Fix NullPointerException when getQuotaUsageInt() invoked
[ https://issues.apache.org/jira/browse/HDFS-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13530: --- Attachment: HDFS-13530.001.patch > NameNode: Fix NullPointerException when getQuotaUsageInt() invoked > -- > > Key: HDFS-13530 > URL: https://issues.apache.org/jira/browse/HDFS-13530 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation, namenode >Reporter: liuhongtong >Priority: Major > Attachments: HDFS-13530.001.patch > > > If the directory is nonexistent, getQuotaUsage rpc call will run into > NullPointerException throwed by > FSDirStatAndListingOp.getQuotaUsageInt() . > I think FSDirStatAndListingOp.getQuotaUsageInt() should throw > FileNotFoundException when the directory is nonexistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13530) NameNode: Fix NullPointerException when getQuotaUsageInt() invoked
liuhongtong created HDFS-13530: -- Summary: NameNode: Fix NullPointerException when getQuotaUsageInt() invoked Key: HDFS-13530 URL: https://issues.apache.org/jira/browse/HDFS-13530 Project: Hadoop HDFS Issue Type: Bug Components: federation, namenode Reporter: liuhongtong If the directory is nonexistent, getQuotaUsage rpc call will run into NullPointerException throwed by FSDirStatAndListingOp.getQuotaUsageInt() . I think FSDirStatAndListingOp.getQuotaUsageInt() should throw FileNotFoundException when the directory is nonexistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1) Remove SCM Block DB
[
https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465328#comment-16465328
]
genericqa commented on HDDS-1:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 35 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
57s{color} | {color:red} hadoop-hdds/common in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} hadoop-hdds/server-scm in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
50s{color} | {color:red} hadoop-ozone/common in trunk has 2 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} hadoop-ozone/ozone-manager in trunk has 1 extant
Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} hadoop-ozone/tools in trunk has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 27m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red} hadoop-hdds/common generated 1 new + 0 unchanged - 1
fixed = 1 total (was 1) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}
[jira] [Commented] (HDDS-1) Remove SCM Block DB
[ https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465297#comment-16465297 ] Xiaoyu Yao commented on HDDS-1: --- Adding patch-3 that addresses the checkstyle/findbugs issues (most of them are existing issues unrelated to this change). > Remove SCM Block DB > --- > > Key: HDDS-1 > URL: https://issues.apache.org/jira/browse/HDDS-1 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-1.002.patch, HDDS-1.003.patch, HDFS-13504.001.patch > > > The block/key information is maintained by Ozone Master (a.k.a. KSM). This > ticket is opened to remove the redundant block db at SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1) Remove SCM Block DB
[ https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-1: -- Attachment: HDDS-1.003.patch > Remove SCM Block DB > --- > > Key: HDDS-1 > URL: https://issues.apache.org/jira/browse/HDDS-1 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-1.002.patch, HDDS-1.003.patch, HDFS-13504.001.patch > > > The block/key information is maintained by Ozone Master (a.k.a. KSM). This > ticket is opened to remove the redundant block db at SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13529) Fix default trash policy emptier trigger time correctly
[
https://issues.apache.org/jira/browse/HDFS-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465291#comment-16465291
]
genericqa commented on HDFS-13529:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 25s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
14s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13529 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922193/HDFS-13529-trunk.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4426648cd520 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e9159db |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24141/testReport/ |
| Max. process+thread count | 1437 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24141/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix default trash policy emptier
[jira] [Commented] (HDFS-13529) Fix default trash policy emptier trigger time correctly
[
https://issues.apache.org/jira/browse/HDFS-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465250#comment-16465250
]
He Xiaoqiao commented on HDFS-13529:
submit v001 patch and trigger jenkins.
> Fix default trash policy emptier trigger time correctly
> ---
>
> Key: HDFS-13529
> URL: https://issues.apache.org/jira/browse/HDFS-13529
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.0, 2.7.6, 3.2.0, 2.9.2, 2.8.5
>Reporter: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-13529-trunk.001.patch
>
>
> Trash emptier is designed to auto trigger at UTC 00:00, however I am confused
> all the time that it usually triggers at a few minutes even half a hour after
> UTC 00:00 actually in our production cluster.
> The main reason is default policy emptier thread sleep more time than as
> expect, since it does not consider the delete operation time cost itself.
> especially for a large cluster, auto trash cleaner may cost dozens of minutes.
> The right way is that gets current time {{now}} before calculate {{end}} time.
> {code:java}
> long now = Time.now();
> while (true) {
> end = ceiling(now, emptierInterval);
> try { // sleep for interval
> Thread.sleep(end - now);
> } catch (InterruptedException e) {
> break; // exit on interrupt
> }
> try {
> now = Time.now();
> .. // delete trash checkpoint
> } catch (Exception e) {
> LOG.warn("RuntimeException during Trash.Emptier.run(): ", e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13529) Fix default trash policy emptier trigger time correctly
[
https://issues.apache.org/jira/browse/HDFS-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Xiaoqiao updated HDFS-13529:
---
Status: Patch Available (was: Open)
> Fix default trash policy emptier trigger time correctly
> ---
>
> Key: HDFS-13529
> URL: https://issues.apache.org/jira/browse/HDFS-13529
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.6, 3.1.0, 3.2.0, 2.9.2, 2.8.5
>Reporter: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-13529-trunk.001.patch
>
>
> Trash emptier is designed to auto trigger at UTC 00:00, however I am confused
> all the time that it usually triggers at a few minutes even half a hour after
> UTC 00:00 actually in our production cluster.
> The main reason is default policy emptier thread sleep more time than as
> expect, since it does not consider the delete operation time cost itself.
> especially for a large cluster, auto trash cleaner may cost dozens of minutes.
> The right way is that gets current time {{now}} before calculate {{end}} time.
> {code:java}
> long now = Time.now();
> while (true) {
> end = ceiling(now, emptierInterval);
> try { // sleep for interval
> Thread.sleep(end - now);
> } catch (InterruptedException e) {
> break; // exit on interrupt
> }
> try {
> now = Time.now();
> .. // delete trash checkpoint
> } catch (Exception e) {
> LOG.warn("RuntimeException during Trash.Emptier.run(): ", e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13529) Fix default trash policy emptier trigger time correctly
[
https://issues.apache.org/jira/browse/HDFS-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Xiaoqiao updated HDFS-13529:
---
Attachment: HDFS-13529-trunk.001.patch
> Fix default trash policy emptier trigger time correctly
> ---
>
> Key: HDFS-13529
> URL: https://issues.apache.org/jira/browse/HDFS-13529
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.0, 2.7.6, 3.2.0, 2.9.2, 2.8.5
>Reporter: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-13529-trunk.001.patch
>
>
> Trash emptier is designed to auto trigger at UTC 00:00, however I am confused
> all the time that it usually triggers at a few minutes even half a hour after
> UTC 00:00 actually in our production cluster.
> The main reason is default policy emptier thread sleep more time than as
> expect, since it does not consider the delete operation time cost itself.
> especially for a large cluster, auto trash cleaner may cost dozens of minutes.
> The right way is that gets current time {{now}} before calculate {{end}} time.
> {code:java}
> long now = Time.now();
> while (true) {
> end = ceiling(now, emptierInterval);
> try { // sleep for interval
> Thread.sleep(end - now);
> } catch (InterruptedException e) {
> break; // exit on interrupt
> }
> try {
> now = Time.now();
> .. // delete trash checkpoint
> } catch (Exception e) {
> LOG.warn("RuntimeException during Trash.Emptier.run(): ", e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13529) Fix default trash policy emptier trigger time correctly
He Xiaoqiao created HDFS-13529:
--
Summary: Fix default trash policy emptier trigger time correctly
Key: HDFS-13529
URL: https://issues.apache.org/jira/browse/HDFS-13529
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.6, 3.1.0, 3.2.0, 2.9.2, 2.8.5
Reporter: He Xiaoqiao
Trash emptier is designed to auto trigger at UTC 00:00, however I am confused
all the time that it usually triggers at a few minutes even half a hour after
UTC 00:00 actually in our production cluster.
The main reason is default policy emptier thread sleep more time than as
expect, since it does not consider the delete operation time cost itself.
especially for a large cluster, auto trash cleaner may cost dozens of minutes.
The right way is that gets current time {{now}} before calculate {{end}} time.
{code:java}
long now = Time.now();
while (true) {
end = ceiling(now, emptierInterval);
try { // sleep for interval
Thread.sleep(end - now);
} catch (InterruptedException e) {
break; // exit on interrupt
}
try {
now = Time.now();
.. // delete trash checkpoint
} catch (Exception e) {
LOG.warn("RuntimeException during Trash.Emptier.run(): ", e);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1) Remove SCM Block DB
[ https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465227#comment-16465227 ] Anu Engineer commented on HDDS-1: - +1, I tested these changes with Acceptance test. I think we should commit the patch-2. Even if there are things that get broken by this patch. Since the core functionality is up, we can commit this refactor and continue fixing broken features before release. We will also deploy this binary based on this patch in a cluster and get some numbers. cc: [~msingh] , [~nandakumar131] , [~elek] > Remove SCM Block DB > --- > > Key: HDDS-1 > URL: https://issues.apache.org/jira/browse/HDDS-1 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-1.002.patch, HDFS-13504.001.patch > > > The block/key information is maintained by Ozone Master (a.k.a. KSM). This > ticket is opened to remove the redundant block db at SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-19) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-4309324-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HDDS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465226#comment-16465226 ] Mukul Kumar Singh commented on HDDS-19: --- Thanks for working on this [~ljain]. The patch looks good to me. +1. > Ozone: Update ozone to latest ratis snapshot build > (0.1.1-alpha-4309324-SNAPSHOT) > - > > Key: HDDS-19 > URL: https://issues.apache.org/jira/browse/HDDS-19 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-19.001.patch, HDFS-13456-HDFS-7240.001.patch, > HDFS-13456-HDFS-7240.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1) Remove SCM Block DB
[
https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465208#comment-16465208
]
genericqa commented on HDDS-1:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 35 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 31m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
0s{color} | {color:red} hadoop-hdds/common in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
44s{color} | {color:red} hadoop-hdds/server-scm in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red} hadoop-ozone/common in trunk has 2 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red} hadoop-ozone/ozone-manager in trunk has 1 extant
Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red} hadoop-ozone/tools in trunk has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 27m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 57s{color} | {color:orange} root: The patch generated 1 new + 2 unchanged -
0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
7s{color} | {color:red} hadoop-hdds/common generated 1 new + 1 unchanged - 0
fixed = 2 total (was 1) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdds/container-service generated 5 new + 0
unchanged - 0 fixed = 5 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
15s{color} | {color:green} the patch
[jira] [Commented] (HDFS-13245) RBF: State store DBMS implementation
[
https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465149#comment-16465149
]
genericqa commented on HDFS-13245:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 5s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
41s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13245 |
| JIRA Patch URL |
[jira] [Commented] (HDDS-20) Ozone: Add support for rename key within a bucket for rpc client
[ https://issues.apache.org/jira/browse/HDDS-20?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465108#comment-16465108 ] Mukul Kumar Singh commented on HDDS-20: --- Thanks for working on the patch [~ljain]. The patch looks really good to me. Please find my comments as following. 1) KeyManagerImpl.java: a) Lets add a check that if the "from" and "to" key names are same then lets return without changing anything in database. Lets add this test after verifying that the from key exists in the database. b) Lets also check that the toKeyName doesn't exists before renaming the key. c) Also lets add a check that if the fromKeyName.length is zero, then lets skip processing of the key name. d) 440: We should have a different error code in place of "FAILED_KEY_NOT_FOUND" 2) ClientProtocol.java:292, "renamed" in place of replaced/ 3) Please also raise jira for supporting renaming for RestClient. 4) Lets also raise another jira to support rename of directory in ozonefs atomically. > Ozone: Add support for rename key within a bucket for rpc client > > > Key: HDDS-20 > URL: https://issues.apache.org/jira/browse/HDDS-20 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-20.001.patch, HDFS-13228-HDFS-7240.001.patch > > > This jira aims to implement rename operation on a key within a bucket for rpc > client. OzoneFilesystem currently rewrites a key on rename. Addition of this > operation would simplify renames in OzoneFilesystem as renames would just be > a db update in ksm. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HDFS-13245: Attachment: HDFS-13245.009.patch > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch, > HDFS-13245.003.patch, HDFS-13245.004.patch, HDFS-13245.005.patch, > HDFS-13245.006.patch, HDFS-13245.007.patch, HDFS-13245.008.patch, > HDFS-13245.009.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HDFS-13245: Status: Patch Available (was: Open) > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch, > HDFS-13245.003.patch, HDFS-13245.004.patch, HDFS-13245.005.patch, > HDFS-13245.006.patch, HDFS-13245.007.patch, HDFS-13245.008.patch, > HDFS-13245.009.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1) Remove SCM Block DB
[
https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465077#comment-16465077
]
genericqa commented on HDDS-1:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 35 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdds/common in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} hadoop-hdds/server-scm in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
50s{color} | {color:red} hadoop-ozone/common in trunk has 2 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} hadoop-ozone/ozone-manager in trunk has 1 extant
Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} hadoop-ozone/tools in trunk has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 27m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 12s{color} | {color:orange} root: The patch generated 1 new + 2 unchanged -
0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
4s{color} | {color:red} hadoop-hdds/common generated 1 new + 1 unchanged - 0
fixed = 2 total (was 1) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
57s{color} | {color:red} hadoop-hdds/container-service generated 5 new + 0
unchanged - 0 fixed = 5 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
11s{color} | {color:green} the patch
[jira] [Commented] (HDDS-17) Add node to container map class to simplify state in SCM
[
https://issues.apache.org/jira/browse/HDDS-17?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465053#comment-16465053
]
genericqa commented on HDDS-17:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 33m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
23s{color} | {color:red} hadoop-hdds/common in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
1s{color} | {color:red} hadoop-hdds/server-scm in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
58s{color} | {color:red} hadoop-ozone/tools in trunk has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 40m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 40m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
2s{color} | {color:red} hadoop-hdds/server-scm generated 3 new + 1 unchanged -
0 fixed = 4 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
59s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
31s{color} | {color:green} server-scm in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
10s{color} | {color:green} tools in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdds/server-scm |
| | Synchronization performed on java.util.concurrent.ConcurrentHashMap in
org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.getContainers(UUID)
At
Node2ContainerMap.java:org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.getContainers(UUID)
At
[jira] [Commented] (HDFS-13403) libhdfs++: Use hdfs::IoService object rather than asio::io_service
[ https://issues.apache.org/jira/browse/HDFS-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465026#comment-16465026 ] Istvan Fajth commented on HDFS-13403: - Hi [~James C], I am wondering if the build_fixes.patch can be committed to the trunk? As I see it is still missing, I just ran into it again :) Thank you in advance! > libhdfs++: Use hdfs::IoService object rather than asio::io_service > -- > > Key: HDFS-13403 > URL: https://issues.apache.org/jira/browse/HDFS-13403 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13403.000.patch, build_fixes.patch > > > At the moment the hdfs::IoService is a simple wrapper over asio's io_service > object. I'd like to make this smarter and have it do things like track which > tasks are queued, validate that dependencies of tasks exist, and monitor > ioservice throughput and contention. In order to get there we need to use > have all components in the library to go through the hdfs::IoService rather > than directly interacting with the asio::io_service. The only time the > asio::io_service should be used is when calling things like asio::async_write > that need an io_service&. HDFS-11884 will be able get rid of those remaining > instances once this work is in place. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1) Remove SCM Block DB
[ https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465005#comment-16465005 ] Xiaoyu Yao commented on HDDS-1: --- Freon and most of the unit tests have been working fine so far with 002 patch. Still some unit tests, SQLCli/SCMCli and BlockDeletion service to validate and test. Post a patch for early review, changes included in this patch: # change to use long for container ID and local ID within the container (for block). # Decouple container from pipeline object. # Remove block db from SCM. > Remove SCM Block DB > --- > > Key: HDDS-1 > URL: https://issues.apache.org/jira/browse/HDDS-1 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-1.002.patch, HDFS-13504.001.patch > > > The block/key information is maintained by Ozone Master (a.k.a. KSM). This > ticket is opened to remove the redundant block db at SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1) Remove SCM Block DB
[ https://issues.apache.org/jira/browse/HDDS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-1: -- Attachment: HDDS-1.002.patch > Remove SCM Block DB > --- > > Key: HDDS-1 > URL: https://issues.apache.org/jira/browse/HDDS-1 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-1.002.patch, HDFS-13504.001.patch > > > The block/key information is maintained by Ozone Master (a.k.a. KSM). This > ticket is opened to remove the redundant block db at SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10290) Move getBlocks calls to DataNode in Balancer
[
https://issues.apache.org/jira/browse/HDFS-10290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465002#comment-16465002
]
xiaoli commented on HDFS-10290:
---
try:https://issues.apache.org/jira/browse/HDFS-13183
> Move getBlocks calls to DataNode in Balancer
>
>
> Key: HDFS-10290
> URL: https://issues.apache.org/jira/browse/HDFS-10290
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: balancer mover
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>Priority: Major
>
> In current implementation, Balancer asks NameNode for a list of blocks on
> specific DataNode. This made workload of NameNode heavier, and actually it
> caused NameNode flappy when average # of blocks on each DataNode reaches
> 1,000,000 (NameNode heap size is 192GB, cpu: Xeon E5-2630 * 2).
> Recently I investigated whether {{getBlocks}} invocation from Balancer can be
> handled by DataNodes, turned out to be practical.
> The only pitfall is: since DataNode has no information about other locations
> of each block it possesses, some block move may fail (since target node may
> already has a replica of that particular block).
> I think this may be beneficial for large clusters.
> Any suggestions or comments?
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
