[jira] [Commented] (HDFS-10460) Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block
[
https://issues.apache.org/jira/browse/HDFS-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345815#comment-15345815
]
Hadoop QA commented on HDFS-10460:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e2f6409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812710/HDFS-10460-05.patch |
| JIRA Issue | HDFS-10460 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 74513bc365ab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 69c21d2 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (HDFS-10328) Add per-cache-pool default replication num configuration
[ https://issues.apache.org/jira/browse/HDFS-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345791#comment-15345791 ] Colin Patrick McCabe commented on HDFS-10328: - Sorry for the breakage, [~kshukla]. HDFS-10555 should have fixed it-- check it out. > Add per-cache-pool default replication num configuration > > > Key: HDFS-10328 > URL: https://issues.apache.org/jira/browse/HDFS-10328 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching >Reporter: xupeng >Assignee: xupeng >Priority: Minor > Fix For: 2.3.0 > > Attachments: HDFS-10328.001.patch, HDFS-10328.002.patch, > HDFS-10328.003.patch, HDFS-10328.004.patch > > > For now, hdfs cacheadmin can not set a default replication num for cached > directive in the same cachepool. Each cache directive added in the same cache > pool should set their own replication num individually. > Consider this situation, we add daily hive table into cache pool "hive" .Each > time i should set the same replication num for every table directive in the > same cache pool. > I think we should enable setting a default replication num for a cachepool > that every cache directive in the pool can inherit replication configuration > from the pool. Also cache directive can override replication configuration > explicitly by calling "add & modify directive -replication" command from > cli. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8940) Support for large-scale multi-tenant inotify service
[ https://issues.apache.org/jira/browse/HDFS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345777#comment-15345777 ] Zhe Zhang commented on HDFS-8940: - Thanks for the insight [~cmccabe]! bq. You would probably want an HA service for fetching HDFS inotify messages You mean reading inotify messages from the SbNN? It's a very attractive idea from scalability angle. But how would we handle the staleness? The SbNN could be a few mins behind ANN right? > Support for large-scale multi-tenant inotify service > > > Key: HDFS-8940 > URL: https://issues.apache.org/jira/browse/HDFS-8940 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ming Ma > Attachments: Large-Scale-Multi-Tenant-Inotify-Service.pdf > > > HDFS-6634 provides the core inotify functionality. We would like to extend > that to provide a large-scale service that ten of thousands of clients can > subscribe to. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10530) BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy
[
https://issues.apache.org/jira/browse/HDFS-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345772#comment-15345772
]
Zhe Zhang commented on HDFS-10530:
--
Thanks [~demongaorui] for reporting this. Yes I believe the right behavior in
the example is to distribute to 9 racks, agreed {{isPlacementPolicySatisfied}}
should use {{getRealTotalBlockNum}}.
A drawback is that this might cause significant more traffic when a new rack is
added. But considering 1) adding a new rack is a rare scenario; 2) we are
throttling reconstruction work, I think we can make this change for better
fault tolerance.
We should add a follow-on task to put this kind of reconstruction tasks to
lowest priority in {{LowRedundancyBlocks}}.
> BlockManager reconstruction work scheduling should correctly adhere to EC
> block placement policy
>
>
> Key: HDFS-10530
> URL: https://issues.apache.org/jira/browse/HDFS-10530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: GAO Rui
>Assignee: GAO Rui
>
> This issue was found by [~tfukudom].
> Under RS-DEFAULT-6-3-64k EC policy,
> 1. Create an EC file, the file was witten to all the 5 racks( 2 dns for each)
> of the cluster.
> 2. Reconstruction work would be scheduled if the 6th rack is added.
> 3. While adding the 7th rack or more racks will not trigger reconstruction
> work.
> Based on default EC block placement policy defined in
> âBlockPlacementPolicyRackFaultTolerant.javaâ, EC file should be able to be
> scheduled to distribute to 9 racks if possible.
> In *BlockManager#isPlacementPolicySatisfied(BlockInfo storedBlock)* ,
> *numReplicas* of striped blocks might should be *getRealTotalBlockNum()*,
> instead of *getRealDataBlockNum()*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8940) Support for large-scale multi-tenant inotify service
[ https://issues.apache.org/jira/browse/HDFS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345766#comment-15345766 ] Colin Patrick McCabe commented on HDFS-8940: I think Kafka would be a great choice for scaling HDFS inotify. You would probably want an HA service for fetching HDFS inotify messages, and then just put them directly into Kafka. No serialization needed because it's already protobuf. > Support for large-scale multi-tenant inotify service > > > Key: HDFS-8940 > URL: https://issues.apache.org/jira/browse/HDFS-8940 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ming Ma > Attachments: Large-Scale-Multi-Tenant-Inotify-Service.pdf > > > HDFS-6634 provides the core inotify functionality. We would like to extend > that to provide a large-scale service that ten of thousands of clients can > subscribe to. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10555) Unable to loadFSEdits due to a failure in readCachePoolInfo
[
https://issues.apache.org/jira/browse/HDFS-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345757#comment-15345757
]
Hudson commented on HDFS-10555:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #10005 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10005/])
HDFS-10555: Unable to loadFSEdits due to a failure in readCachePoolInfo.
(uma.gangumalla: rev 01643020d24b606c26cb2dcaefdb6e47a5f06c8f)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
> Unable to loadFSEdits due to a failure in readCachePoolInfo
> ---
>
> Key: HDFS-10555
> URL: https://issues.apache.org/jira/browse/HDFS-10555
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Critical
> Attachments: HDFS-10555-00.patch
>
>
> Recently some tests are failing and unable to loadFSEdits due to a failure in
> readCachePoolInfo.
> Here in below code
> FSImageSerialization.java
> {code}
> }
> if ((flags & ~0x2F) != 0) {
> throw new IOException("Unknown flag in CachePoolInfo: " + flags);
> }
> {code}
> When all values of CachePool variable set to true, flags value & ~0x2F turns
> out to non zero value. So, this condition failing due to the addition of 0x20
> and changing value from ~0x1F to ~0x2F.
> May be to fix this issue, we may can change multiply value to ~0x3F
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10460) Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block
[
https://issues.apache.org/jira/browse/HDFS-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-10460:
Attachment: HDFS-10460-05.patch
Attached another patch fixing checkstyle warning. [~drankye], kindly review
the latest patch again.
> Erasure Coding: Recompute block checksum for a particular range less than
> file size on the fly by reconstructing missed block
> -
>
> Key: HDFS-10460
> URL: https://issues.apache.org/jira/browse/HDFS-10460
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-10460-00.patch, HDFS-10460-01.patch,
> HDFS-10460-02.patch, HDFS-10460-03.patch, HDFS-10460-04.patch,
> HDFS-10460-05.patch
>
>
> This jira is HDFS-9833 follow-on task to address reconstructing block and
> then recalculating block checksum for a particular range query.
> For example,
> {code}
> // create a file 'stripedFile1' with fileSize = cellSize * numDataBlocks =
> 65536 * 6 = 393216
> FileChecksum stripedFileChecksum = getFileChecksum(stripedFile1, 10, true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10460) Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block
[
https://issues.apache.org/jira/browse/HDFS-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345701#comment-15345701
]
Hadoop QA commented on HDFS-10460:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
114 unchanged - 0 fixed = 115 total (was 114) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 24s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e2f6409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812595/HDFS-10460-04.patch |
| JIRA Issue | HDFS-10460 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux d4deea679ccc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 99e5dd6 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-8940) Support for large-scale multi-tenant inotify service
[
https://issues.apache.org/jira/browse/HDFS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345544#comment-15345544
]
Zhe Zhang commented on HDFS-8940:
-
This is a very useful optimization. Thanks [~mingma] for proposing the ideas
and design.
Just wanted to quickly follow-up and see if there's any interest in continuing
the work? We are facing a similar issue where NN suffers from large listing
operations.
Based on some initial requirements collection, I think in our environment Kafka
is a better fit. This is because some workloads do need to monitor for a
potentially very large number of events. E.g. ETL service monitoring all newly
generated files under a subtree and synching them to another cluster. If
dependency is a concern, maybe this optimization can be a standalone project
which depends on both HDFS and Kafka? It seems the current design uses inotify
APIs in a pretty clean way.
We should also think of a way to quantify the potential benefit -- e.g.
assuming we have the audit log from a large cluster, how to estimate the
reduced locking contention if all listing ops are eliminated (and inotify ops
added)? IIUC inotify calls don't {{readLock}} the namespace.
> Support for large-scale multi-tenant inotify service
>
>
> Key: HDFS-8940
> URL: https://issues.apache.org/jira/browse/HDFS-8940
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
> Attachments: Large-Scale-Multi-Tenant-Inotify-Service.pdf
>
>
> HDFS-6634 provides the core inotify functionality. We would like to extend
> that to provide a large-scale service that ten of thousands of clients can
> subscribe to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345518#comment-15345518
]
Anu Engineer commented on HDFS-10560:
-
[~linyiqun] Thank you for finding this issue and providing a fix for the issue.
Overall the patch looks very good. I had some minor comments.
I see that sometimes we use
{code:title=DiskBalancerWorkItem.java|borderStyle=solid}
return MAPPER.writeValueAsString(this);
{code}
but in other places we use
{code:title=DiskBalancer.java|borderStyle=solid}
return JsonUtil.toJsonString(pathMap);
{code}
If this is not intentional can we please use a consistent way of doing it. I
don't have any preference, either way is fine.
{{DiskBalancerWorkStatus.java#currentStateString}} Since we enable
INDENT_OUTPUT here, was wondering is that going to carry over to
{{toJsonString}}. Not that it is an error or anything, just making sure that
side-effect is intentional.
{{CombinedHostsFileReader.java}} and {{CombinedHostsFileWriter.java}} are not
part of diskbalancer. Should we clean up those files in another JIRA ?
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10560-HDFS-1312.001.patch,
> HDFS-10560-HDFS-1312.002.patch
>
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10220) A large number of expired leases can make namenode unresponsive and cause failover
[ https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345509#comment-15345509 ] Zhe Zhang commented on HDFS-10220: -- Agreed. HDFS-6757 made a pretty big change which is hard to get around. Thanks Ravi for backporting to 2.7. > A large number of expired leases can make namenode unresponsive and cause > failover > -- > > Key: HDFS-10220 > URL: https://issues.apache.org/jira/browse/HDFS-10220 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Nicolas Fraison >Assignee: Nicolas Fraison > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HADOOP-10220.001.patch, HADOOP-10220.002.patch, > HADOOP-10220.003.patch, HADOOP-10220.004.patch, HADOOP-10220.005.patch, > HADOOP-10220.006.patch, HADOOP-10220.007.patch, threaddump_zkfc.txt > > > I have faced a namenode failover due to unresponsive namenode detected by the > zkfc with lot's of WARN messages (5 millions) like this one: > _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All > existing blocks are COMPLETE, lease removed, file closed._ > On the threaddump taken by the zkfc there are lots of thread blocked due to a > lock. > Looking at the code, there are a lock taken by the LeaseManager.Monitor when > some lease must be released. Due to the really big number of lease to be > released the namenode has taken too many times to release them blocking all > other tasks and making the zkfc thinking that the namenode was not > available/stuck. > The idea of this patch is to limit the number of leased released each time we > check for lease so the lock won't be taken for a too long time period. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-1312) Re-balance disks within a Datanode
[ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-1312: --- Attachment: HDFS-1312.004.patch Posting another merge patch against trunk. > Re-balance disks within a Datanode > -- > > Key: HDFS-1312 > URL: https://issues.apache.org/jira/browse/HDFS-1312 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode >Reporter: Travis Crawford >Assignee: Anu Engineer > Attachments: Architecture_and_test_update.pdf, > Architecture_and_testplan.pdf, HDFS-1312.001.patch, HDFS-1312.002.patch, > HDFS-1312.003.patch, HDFS-1312.004.patch, disk-balancer-proposal.pdf > > > Filing this issue in response to ``full disk woes`` on hdfs-user. > Datanodes fill their storage directories unevenly, leading to situations > where certain disks are full while others are significantly less used. Users > at many different sites have experienced this issue, and HDFS administrators > are taking steps like: > - Manually rebalancing blocks in storage directories > - Decomissioning nodes & later readding them > There's a tradeoff between making use of all available spindles, and filling > disks at the sameish rate. Possible solutions include: > - Weighting less-used disks heavier when placing new blocks on the datanode. > In write-heavy environments this will still make use of all spindles, > equalizing disk use over time. > - Rebalancing blocks locally. This would help equalize disk use as disks are > added/replaced in older cluster nodes. > Datanodes should actively manage their local disk so operator intervention is > not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10552) DiskBalancer "-query" results in NPE if no plan for the node
[
https://issues.apache.org/jira/browse/HDFS-10552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-10552:
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the reviews. I have committed this to the feature branch
> DiskBalancer "-query" results in NPE if no plan for the node
>
>
> Key: HDFS-10552
> URL: https://issues.apache.org/jira/browse/HDFS-10552
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: HDFS-1312
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10552-HDFS-1312.001.patch,
> HDFS-10552-HDFS-1312.002.patch, HDFS-10552-HDFS-1312.003.patch,
> HDFS-10552-HDFS-1312.004.patch
>
>
> {code}
> 16/06/20 11:50:16 INFO command.Command: Executing "query plan" command.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$QueryPlanStatusResponseProto$Builder.setPlanID(ClientDatanodeProtocolProtos.java:12782)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.queryDiskBalancerPlan(ClientDatanodeProtocolServerSideTranslatorPB.java:340)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:17513)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-10551:
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thank you all for the reviews. I have committed this to the feature branch.
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch, HDFS-10551-HDFS-1312.003.patch,
> HDFS-10551-HDFS-1312.004.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10473) Allow only suitable storage policies to be set on striped files
[ https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345455#comment-15345455 ] Zhe Zhang commented on HDFS-10473: -- Thanks [~umamaheswararao] and [~jingzhao]! Just caught up on this, agree the latest fix is a better fix than the one I +1'ed on. > Allow only suitable storage policies to be set on striped files > --- > > Key: HDFS-10473 > URL: https://issues.apache.org/jira/browse/HDFS-10473 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-10473-01.patch, HDFS-10473-02.patch, > HDFS-10473-03.patch, HDFS-10473-04.patch, HDFS-10473-05.patch > > > Currently some of existing storage policies are not suitable for striped > layout files. > This JIRA proposes to reject setting storage policy on striped files. > Another thought is to allow only suitable storage polices like ALL_SSD. > Since the major use case of EC is for cold data, this may not be at high > importance. So, I am ok to reject setting storage policy on striped files at > this stage. Please suggest if others have some thoughts on this. > Thanks [~zhz] for offline discussion on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345450#comment-15345450
]
Hadoop QA commented on HDFS-10551:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c88012f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812637/HDFS-10551-HDFS-1312.004.patch
|
| JIRA Issue | HDFS-10551 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux cfdbedff3c26 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-1312 / 4f5beb6 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15880/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15880/console |
| Powered by | Apache
[jira] [Commented] (HDFS-3599) Better expose when under-construction files are preventing DN decommission
[
https://issues.apache.org/jira/browse/HDFS-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345422#comment-15345422
]
Wei-Chiu Chuang commented on HDFS-3599:
---
I think HDFS-8344 will fix this issue by forcing the lease to time out.
> Better expose when under-construction files are preventing DN decommission
> --
>
> Key: HDFS-3599
> URL: https://issues.apache.org/jira/browse/HDFS-3599
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Todd Lipcon
>Assignee: Zhe Zhang
>
> Filing on behalf of Konstantin Olchanski:
> {quote}
> I have been trying to decommission a data node, but the process
> stalled. I followed the correct instructions, observed my node
> listed in "Decommissioning Nodes", etc, observed "Under Replicated Blocks"
> decrease, etc. But the count went down to "1" and the decommissin process
> stalled.
> There was no visible activity anywhere, nothing was happening (well,
> maybe in some hidden log file somewhere something complained,
> but I did not look).
> It turns out that I had some files stuck in "OPENFORWRITE" mode,
> as reported by "hdfs fsck / -openforwrite -files -blocks -locations -racks":
> {code}
> /users/trinat/data/.fuse_hidden177e0002 0 bytes, 0 block(s),
> OPENFORWRITE: OK
> /users/trinat/data/.fuse_hidden178d0003 0 bytes, 0 block(s),
> OPENFORWRITE: OK
> /users/trinat/data/.fuse_hidden1da30004 0 bytes, 1 block(s),
> OPENFORWRITE: OK
> 0.
> BP-88378204-142.90.119.126-1340494203431:blk_6980480609696383665_20259{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[142.90.111.72:50010|RBW],
> ReplicaUnderConstruction[142.90.119.162:50010|RBW],
> ReplicaUnderConstruction[142.90.119.126:50010|RBW]]} len=0 repl=3
> [/detfac/142.90.111.72:50010, /isac2/142.90.119.162:50010,
> /isac2/142.90.119.126:50010]
> {code}
> After I deleted those files, the decommission process completed successfully.
> Perhaps one can add some visible indication somewhere on the HDFS status web
> page
> that the decommission process is stalled and maybe report why it is stalled?
> Maybe the number of "OPENFORWRITE" files should be listed on the status page
> next to the "Number of Under-Replicated Blocks"? (Since I know that nobody is
> writing
> to my HDFS, the non-zero count would give me a clue that something is wrong).
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10534) NameNode WebUI should display DataNode usage rate with a certain percentile
[
https://issues.apache.org/jira/browse/HDFS-10534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345386#comment-15345386
]
Zhe Zhang commented on HDFS-10534:
--
Thanks [~lewuathe]! Latest patch LGTM overall. A few minors:
# Can we add a sanity check when loading the
{{dfs.namenode.metrics.node-usage.percentile}} config? Assert that it is
between 0 and 1.
# Minor suggestion on the description of
{{dfs.namenode.metrics.node-usage.percentile}} below. This could be confusing
so I think we should be a little more specific.
{code}
This setting specifies the percentile level to report node usage metrics. For
example 0.95 means reporting the node usage for the 95th percentile of all
DataNodes. If this setting is at 0.95 and the reported node usage is 70%, it
means 95% of DataNodes have a usage below 70%.
{code}
# To be consistent with the mathematical [definition |
https://en.wikipedia.org/wiki/Percentile], maybe we should also called it xth
percentile on the html.
{code}
percentileInfo.put("name", StringUtils.format("%dth percentile",
{code}
I'm trying to start a NameNode with the patch to verify the new HTML. If you
have one ready it'd be great to add a screenshot to the JIRA.
Thanks for the nice work!
> NameNode WebUI should display DataNode usage rate with a certain percentile
> ---
>
> Key: HDFS-10534
> URL: https://issues.apache.org/jira/browse/HDFS-10534
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, ui
>Reporter: Zhe Zhang
>Assignee: Kai Sasaki
> Attachments: HDFS-10534.01.patch, HDFS-10534.02.patch,
> HDFS-10534.03.patch, HDFS-10534.04.patch
>
>
> In addition of *Min/Median/Max*, another meaningful metric for cluster
> balance is DN usage rate at a certain percentile (e.g. 90 or 95). We should
> add a config option, and another filed on NN WebUI, to display this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10328) Add per-cache-pool default replication num configuration
[
https://issues.apache.org/jira/browse/HDFS-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345320#comment-15345320
]
Kuhu Shukla commented on HDFS-10328:
[~xupeng], [~cmccabe], it seems that {{TestRetryCacheWithHA ,
TestNamenodeRetryCache, TestCacheDirectives, TestOfflineEditsViewer}}
are failing after this commit. Should I open separate JIRA for these? Thanks!
> Add per-cache-pool default replication num configuration
>
>
> Key: HDFS-10328
> URL: https://issues.apache.org/jira/browse/HDFS-10328
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: caching
>Reporter: xupeng
>Assignee: xupeng
>Priority: Minor
> Fix For: 2.3.0
>
> Attachments: HDFS-10328.001.patch, HDFS-10328.002.patch,
> HDFS-10328.003.patch, HDFS-10328.004.patch
>
>
> For now, hdfs cacheadmin can not set a default replication num for cached
> directive in the same cachepool. Each cache directive added in the same cache
> pool should set their own replication num individually.
> Consider this situation, we add daily hive table into cache pool "hive" .Each
> time i should set the same replication num for every table directive in the
> same cache pool.
> I think we should enable setting a default replication num for a cachepool
> that every cache directive in the pool can inherit replication configuration
> from the pool. Also cache directive can override replication configuration
> explicitly by calling "add & modify directive -replication" command from
> cli.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10567) Improve plan command help message
Lei (Eddy) Xu created HDFS-10567:
Summary: Improve plan command help message
Key: HDFS-10567
URL: https://issues.apache.org/jira/browse/HDFS-10567
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Lei (Eddy) Xu
{code}
--bandwidth Maximum disk bandwidth to be consumed by
diskBalancer. e.g. 10
--maxerror Describes how many errors can be
tolerated while copying between a pair
of disks.
--outFile to write output to, if not
specified defaults will be used.
--plan creates a plan for datanode.
--thresholdPercentagePercentage skew that wetolerate before
diskbalancer starts working e.g. 10
--v Print out the summary of the plan on
console
{code}
We should
* Put the unit into {{--bandwidth}}, or its help message. Is it an integer or
float / double number? Not clear in CLI message.
* Give more details about {{--plan}}. It is not clear what the {{}} is for.
* {{--thresholdPercentage}}, has typo {{wetolerate}} in the error message.
Also it needs to indicated that it is the difference between space utilization
between two disks / volumes. Is it an integer or float / double number?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345310#comment-15345310
]
Mingliang Liu commented on HDFS-10556:
--
Thanks [~jingzhao] for the review and commit!
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-10556-branch-2.000.patch,
> HDFS-10556-branch-2.001.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-10551:
Attachment: HDFS-10551-HDFS-1312.004.patch
Rebase to top of the tree
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch, HDFS-10551-HDFS-1312.003.patch,
> HDFS-10551-HDFS-1312.004.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-10556:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I've committed this into branch-2 and branch-2.8.
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-10556-branch-2.000.patch,
> HDFS-10556-branch-2.001.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-10556:
-
Component/s: distcp
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-10556-branch-2.000.patch,
> HDFS-10556-branch-2.001.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345279#comment-15345279
]
Jing Zhao commented on HDFS-10556:
--
Thanks for the fix, Mingliang. The patch looks good to me. +1.
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10556-branch-2.000.patch,
> HDFS-10556-branch-2.001.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10473) Allow only suitable storage policies to be set on striped files
[ https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10473: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha1 Status: Resolved (was: Patch Available) > Allow only suitable storage policies to be set on striped files > --- > > Key: HDFS-10473 > URL: https://issues.apache.org/jira/browse/HDFS-10473 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-10473-01.patch, HDFS-10473-02.patch, > HDFS-10473-03.patch, HDFS-10473-04.patch, HDFS-10473-05.patch > > > Currently some of existing storage policies are not suitable for striped > layout files. > This JIRA proposes to reject setting storage policy on striped files. > Another thought is to allow only suitable storage polices like ALL_SSD. > Since the major use case of EC is for cold data, this may not be at high > importance. So, I am ok to reject setting storage policy on striped files at > this stage. Please suggest if others have some thoughts on this. > Thanks [~zhz] for offline discussion on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345258#comment-15345258 ] John Zhuge commented on HDFS-6962: -- [~cnauroth], Thanks for the info. I will work with the existing test classes as much as possible. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10514) Augment QueryDiskBalancerPlan to return storage id/type of source/dest volumes
[ https://issues.apache.org/jira/browse/HDFS-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345248#comment-15345248 ] Xiaobing Zhou commented on HDFS-10514: -- The failure testCompareConfigurationClassAgainstXml has been fixed by HDFS-10551. The other two are not related to the patch. > Augment QueryDiskBalancerPlan to return storage id/type of source/dest volumes > -- > > Key: HDFS-10514 > URL: https://issues.apache.org/jira/browse/HDFS-10514 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10514-HDFS-1312.000.patch, > HDFS-10514-HDFS-1312.001.patch > > > DiskBalancerWorkEntry returned by QueryDiskBalancerPlan only contains paths > of source/dest volumes. It's preferable to get storage id/storage type too. > Scan command could show a rich set of information how data is moved between > different volumes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345249#comment-15345249
]
Chris Nauroth commented on HDFS-6962:
-
[~jzhuge], I know {{FSAclBaseTest}} has test cases that cover inheritance of
default ACLs to newly created files and sub-directories. Possibly the other
suites you mentioned would have test cases for default ACL inheritance too. If
those are what you're looking for, then you might not need a new
{{TestAclInheritance}} suite.
Sorry for my delay in reviewing this more deeply. I will aim for no later than
next week to take a closer look.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10566) Submit plan request should throw exception if Datanode is undergoing an upgrade.
[ https://issues.apache.org/jira/browse/HDFS-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reassigned HDFS-10566: Assignee: Xiaobing Zhou > Submit plan request should throw exception if Datanode is undergoing an > upgrade. > - > > Key: HDFS-10566 > URL: https://issues.apache.org/jira/browse/HDFS-10566 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Jitendra Nath Pandey >Assignee: Xiaobing Zhou > > If datanode is in upgrade, it might be simpler to just refuse balancing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10559) DiskBalancer: Use SHA1 for Plan ID
[ https://issues.apache.org/jira/browse/HDFS-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reassigned HDFS-10559: Assignee: Xiaobing Zhou > DiskBalancer: Use SHA1 for Plan ID > -- > > Key: HDFS-10559 > URL: https://issues.apache.org/jira/browse/HDFS-10559 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Xiaobing Zhou >Priority: Trivial > Labels: newbie > > We should use SHA1 instead of Sha512 as the plan id. Since it is much shorter > and easier to handle. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10558) DiskBalancer: Print the full path to plan file
[ https://issues.apache.org/jira/browse/HDFS-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reassigned HDFS-10558: Assignee: Xiaobing Zhou > DiskBalancer: Print the full path to plan file > --- > > Key: HDFS-10558 > URL: https://issues.apache.org/jira/browse/HDFS-10558 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Xiaobing Zhou >Priority: Minor > Labels: newbie > > We should print the full path to plan file when plan command is being run. > That makes it easy to give that path to -execute option. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345214#comment-15345214
]
John Zhuge edited comment on HDFS-6962 at 6/22/16 9:44 PM:
---
[~cnauroth] Are there automated tests for section {{Default Acls}} in test plan
{{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new
section in test plan to cover ACL inheritance. Currently these are ACL related
tests:
TestAcl,TestAclCommands,TestAclCLI,TestViewFileSystemWithAcls,TestViewFsWithAcls,FSAclBaseTest,TestAclWithSnapshot,TestAclConfigFlag,TestAclTransformation,TestFileContextAcl,TestFSImageWithAcl,TestNameNodeAcl,TestAclsEndToEnd,TestOfflineImageViewerForAcl,TestWebHDFSAcl.
Will add a new test class {{namenode/TestAclInheritance}}.
was (Author: jzhuge):
[~cnauroth] Are there automated tests for section {{Default Acls}} in test plan
{{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new
section in test plan to cover ACL inheritance. Will add a new test class
{{namenode/TestAclInheritance}}.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345214#comment-15345214
]
John Zhuge commented on HDFS-6962:
--
[~cnauroth] Are there automated tests for section {{Default Acls}} in test plan
{{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new
section in test plan to cover ACL inheritance. Will add a new test class
{{namenode/TestAclInheritance}}.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10566) Submit plan request should throw exception if Datanode is undergoing an upgrade.
[ https://issues.apache.org/jira/browse/HDFS-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-10566: Description: If datanode is in upgrade, it might be simpler to just refuse balancing. > Submit plan request should throw exception if Datanode is undergoing an > upgrade. > - > > Key: HDFS-10566 > URL: https://issues.apache.org/jira/browse/HDFS-10566 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Jitendra Nath Pandey > > If datanode is in upgrade, it might be simpler to just refuse balancing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10566) Submit plan request should throw exception if Datanode is undergoing an upgrade.
Jitendra Nath Pandey created HDFS-10566: --- Summary: Submit plan request should throw exception if Datanode is undergoing an upgrade. Key: HDFS-10566 URL: https://issues.apache.org/jira/browse/HDFS-10566 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10562) DiskBalancer: update documentation on how to report issues and debug
[
https://issues.apache.org/jira/browse/HDFS-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345202#comment-15345202
]
Hadoop QA commented on HDFS-10562:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 51 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c88012f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812620/HDFS-10562-HDFS-1312.001.patch
|
| JIRA Issue | HDFS-10562 |
| Optional Tests | asflicense mvnsite |
| uname | Linux 248141b1e2be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-1312 / 4f5beb6 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/15879/artifact/patchprocess/whitespace-eol.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15879/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DiskBalancer: update documentation on how to report issues and debug
>
>
> Key: HDFS-10562
> URL: https://issues.apache.org/jira/browse/HDFS-10562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: HDFS-1312
>
> Attachments: HDFS-10562-HDFS-1312.001.patch
>
>
> Add a section in the diskbalancer documentation on how to report issues and
> how to debug diskbalancer usage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345177#comment-15345177
]
Hadoop QA commented on HDFS-10556:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated
23 new + 79 unchanged - 0 fixed = 102 total (was 79) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
17s{color} | {color:green} hadoop-distcp in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:d1c475d |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812607/HDFS-10556-branch-2.001.patch
|
| JIRA Issue | HDFS-10556 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 74973bcca4cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 81497a4 |
| Default Java | 1.7.0_101 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91
[jira] [Updated] (HDFS-10562) DiskBalancer: update documentation on how to report issues and debug
[ https://issues.apache.org/jira/browse/HDFS-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10562: Attachment: HDFS-10562-HDFS-1312.001.patch > DiskBalancer: update documentation on how to report issues and debug > > > Key: HDFS-10562 > URL: https://issues.apache.org/jira/browse/HDFS-10562 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Minor > Fix For: HDFS-1312 > > Attachments: HDFS-10562-HDFS-1312.001.patch > > > Add a section in the diskbalancer documentation on how to report issues and > how to debug diskbalancer usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10562) DiskBalancer: update documentation on how to report issues and debug
[ https://issues.apache.org/jira/browse/HDFS-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10562: Status: Patch Available (was: Open) > DiskBalancer: update documentation on how to report issues and debug > > > Key: HDFS-10562 > URL: https://issues.apache.org/jira/browse/HDFS-10562 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Minor > Fix For: HDFS-1312 > > Attachments: HDFS-10562-HDFS-1312.001.patch > > > Add a section in the diskbalancer documentation on how to report issues and > how to debug diskbalancer usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10565) Erasure Coding: Document about the current allowed storage policies for EC Striped mode files
Uma Maheswara Rao G created HDFS-10565: -- Summary: Erasure Coding: Document about the current allowed storage policies for EC Striped mode files Key: HDFS-10565 URL: https://issues.apache.org/jira/browse/HDFS-10565 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Affects Versions: 3.0.0-alpha1 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G HDFS-10473 implemented to allow only ALL_SSD, HOT, COLD policies to take effect while moving/placing blocks for Striped EC files. This is JIRA to track the documentation about the behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345104#comment-15345104
]
Mingliang Liu commented on HDFS-10556:
--
Per-offline discussion with [~jingzhao], we should keep the {{validate()}}
method signature as it-was. The reason was that it's a public API and
downstream applications may call it directly before calling setters. The v1
patch is to make the code similar to pre- [HDFS-10397]. The changes are:
# Ignore the {{delete}} option if {{useDiff}} is set. This is processed in both
{{setDeleteMissing()}} and {{setUseDiff()}}
# Reuse the {{DistCpOptions}} from v0 patch for unit test
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10556-branch-2.000.patch,
> HDFS-10556-branch-2.001.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10556:
-
Attachment: HDFS-10556-branch-2.001.patch
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10556-branch-2.000.patch,
> HDFS-10556-branch-2.001.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10543) hdfsRead read stops at block boundary
[
https://issues.apache.org/jira/browse/HDFS-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345061#comment-15345061
]
James Clampffer commented on HDFS-10543:
New patch looks good other than two minor issues.
The extra warning "JDK v1.8.0_91 generated 1 new + 29 unchanged - 0 fixed = 30
total (was 29)" that got introduced in this patch was a signed < unsigned
comparison. Those have started to break things (HDFS-10554), could you take a
look at that? Good to know the compiler is picking up on this in some places.
The rest of those warnings are coming from our third-party libraries and I'm
hoping to clean those up soon.
{code}
while (*nbyte != 0 && offset < file_info_->file_length_) {
{code}
I hate to complain about code format, but could you make sure you use two
spaces for tabs? No rationale for this other than keeping consistent with the
rest of the library.
{code}
if(!stat.ok()) {
-return stat;
+ return stat;
}
{code}
> hdfsRead read stops at block boundary
> -
>
> Key: HDFS-10543
> URL: https://issues.apache.org/jira/browse/HDFS-10543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Xiaowei Zhu
>Assignee: James Clampffer
> Attachments: HDFS-10543.HDFS-8707.000.patch,
> HDFS-10543.HDFS-8707.001.patch, HDFS-10543.HDFS-8707.002.patch,
> HDFS-10543.HDFS-8707.003.patch
>
>
> Reproducer:
> char *buf2 = new char[file_info->mSize];
> memset(buf2, 0, (size_t)file_info->mSize);
> int ret = hdfsRead(fs, file, buf2, file_info->mSize);
> delete [] buf2;
> if(ret != file_info->mSize) {
> std::stringstream ss;
> ss << "tried to read " << file_info->mSize << " bytes. but read " <<
> ret << " bytes";
> ReportError(ss.str());
> hdfsCloseFile(fs, file);
> continue;
> }
> When it runs with a file ~1.4GB large, it will return an error like "tried to
> read 146890 bytes. but read 134217728 bytes". The HDFS cluster it runs
> against has a block size of 134217728 bytes. So it seems hdfsRead will stop
> at a block boundary. Looks like a regression. We should add retry to continue
> reading cross blocks in case of files w/ multiple blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10460) Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block
[
https://issues.apache.org/jira/browse/HDFS-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345063#comment-15345063
]
Rakesh R commented on HDFS-10460:
-
Thanks [~drankye], Attached new patch fixing the comments.
bq. 1. In StripedBlockChecksumReconstructor#reconstruct(), could we move the
most of the new codes to a new function like
checksumWithTargetOutput(targetBuffer)? Doing this would separate the checksum
logic from the reconstruction stuffs and when we add to handle more failures
than one, the new function can be reused.
Agreed, I have moved most of the code to new method. Probably will refine it
while implementing multiple failures HDFS-10461
bq. 1) The 1 requestedLen case is good to have, might also have a larger one
like 10?
Added this case also.
bq. 2) Don't quite get why we have an array for the only one filepath. 3) You
have quite a few tests like this, guess you could move the codes of preparing
for the test file to a common place.
I've combined these two points, just modified {{#prepareTestFiles()}} by taking
{{filePaths}} argument and preparing required files for each test case.
> Erasure Coding: Recompute block checksum for a particular range less than
> file size on the fly by reconstructing missed block
> -
>
> Key: HDFS-10460
> URL: https://issues.apache.org/jira/browse/HDFS-10460
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-10460-00.patch, HDFS-10460-01.patch,
> HDFS-10460-02.patch, HDFS-10460-03.patch, HDFS-10460-04.patch
>
>
> This jira is HDFS-9833 follow-on task to address reconstructing block and
> then recalculating block checksum for a particular range query.
> For example,
> {code}
> // create a file 'stripedFile1' with fileSize = cellSize * numDataBlocks =
> 65536 * 6 = 393216
> FileChecksum stripedFileChecksum = getFileChecksum(stripedFile1, 10, true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10555) Unable to loadFSEdits due to a failure in readCachePoolInfo
[
https://issues.apache.org/jira/browse/HDFS-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345054#comment-15345054
]
Hadoop QA commented on HDFS-10555:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e2f6409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812361/HDFS-10555-00.patch |
| JIRA Issue | HDFS-10555 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e44c52c15e4b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 17eae9e |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15876/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15876/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15876/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Unable to loadFSEdits due to a failure in readCachePoolInfo
>
[jira] [Updated] (HDFS-10460) Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block
[
https://issues.apache.org/jira/browse/HDFS-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-10460:
Attachment: HDFS-10460-04.patch
> Erasure Coding: Recompute block checksum for a particular range less than
> file size on the fly by reconstructing missed block
> -
>
> Key: HDFS-10460
> URL: https://issues.apache.org/jira/browse/HDFS-10460
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-10460-00.patch, HDFS-10460-01.patch,
> HDFS-10460-02.patch, HDFS-10460-03.patch, HDFS-10460-04.patch
>
>
> This jira is HDFS-9833 follow-on task to address reconstructing block and
> then recalculating block checksum for a particular range query.
> For example,
> {code}
> // create a file 'stripedFile1' with fileSize = cellSize * numDataBlocks =
> 65536 * 6 = 393216
> FileChecksum stripedFileChecksum = getFileChecksum(stripedFile1, 10, true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10563) Block reports could be silently dropped by NN
[ https://issues.apache.org/jira/browse/HDFS-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-10563: - Priority: Critical (was: Major) > Block reports could be silently dropped by NN > - > > Key: HDFS-10563 > URL: https://issues.apache.org/jira/browse/HDFS-10563 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0, 2.7.3 >Reporter: Staffan Friberg >Priority: Critical > > Reading through the block reporting code I think I've spotted a case when > block reports can silently be dropped and leave thread waiting indefinitely > on a FutureTask that never will be executed. > The BlockReportProcessingThread.enqueue method doesn't return any status on > if the enqueuing of the task was successful and does not handle the case when > the queue is full and offer return false. > Going back through the call stack to BlockManager.runBlockOp, which > indirectly calls enqueue with a FutureTask and then proceeds to do get() om > the task. > So if the internal queue in the BlockReportingProcessingThread is full, the > BR would never be handled and the thread queuing the task would wait > indefinitely on the FutureTask that will never be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10563) Block reports could be silently dropped by NN
[ https://issues.apache.org/jira/browse/HDFS-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-10563: - Affects Version/s: (was: 3.0.0-beta1) 2.7.3 2.8.0 > Block reports could be silently dropped by NN > - > > Key: HDFS-10563 > URL: https://issues.apache.org/jira/browse/HDFS-10563 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0, 2.7.3 >Reporter: Staffan Friberg >Priority: Critical > > Reading through the block reporting code I think I've spotted a case when > block reports can silently be dropped and leave thread waiting indefinitely > on a FutureTask that never will be executed. > The BlockReportProcessingThread.enqueue method doesn't return any status on > if the enqueuing of the task was successful and does not handle the case when > the queue is full and offer return false. > Going back through the call stack to BlockManager.runBlockOp, which > indirectly calls enqueue with a FutureTask and then proceeds to do get() om > the task. > So if the internal queue in the BlockReportingProcessingThread is full, the > BR would never be handled and the thread queuing the task would wait > indefinitely on the FutureTask that will never be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10564) UNDER MIN REPL'D BLOCKS should be prioritized for replication
Elliott Clark created HDFS-10564: Summary: UNDER MIN REPL'D BLOCKS should be prioritized for replication Key: HDFS-10564 URL: https://issues.apache.org/jira/browse/HDFS-10564 Project: Hadoop HDFS Issue Type: Improvement Reporter: Elliott Clark When datanodes get drained they are probably being drained because the hardware is bad, or suspect. The blocks that have no live nodes should be prioritized. However it appears not to be the case at all. Draining full nodes with lots of blocks but only a handful of under min replicated blocks takes about the full time before fsck reports clean again. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345017#comment-15345017
]
Mingliang Liu commented on HDFS-10551:
--
[~anu] I've no strong preference about the above comments. You can feel free to
choose which one is more suitable to the current code base. Thanks.
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch, HDFS-10551-HDFS-1312.003.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344990#comment-15344990
]
Anu Engineer commented on HDFS-10551:
-
[~liuml07] Thanks for your comments.
bq. In Command constructor, // These arguments are valid for all commands. can
be deleted.
I left there for the future maintainers. Earlier we had a command like -help on
each command, but now we have moved it to an independent -help command.
So you have no reference to how to use this. I think this comment helps with
someone how wants to add an option to all commands in disk balancer. please let
me know if you disagree and we must remove this comment.
bq. In TestDiskBalancerCommand other test cases can also use the class filed
conf (e.g. testReadClusterFromJson()). This will also address the checkstyle
warning I think.
Agreed, the latest patch fixed that issue and we don't have any more checkstyle
warnings
bq. For the config key name dfs.disk.balancer.max.disk.throughputInMBperSec,
I'd suggest dfs.disk.balancer.max.disk.bandwidthPerSec. I think throughput and
bandwidth have different meanings in different context and bandwidth here is
better. This also conforms to existing names like
dfs.image.transfer.bandwidthPerSec, dfs.balance.bandwidthPerSec and
dfs.datanode.balance.bandwidthPerSec, to name a few.
if my memory serves me correctly it was named something like that because a
code review comment wanted to make sure that Units are expressed as part of the
name. Names like bandwithPerSec does not tell you if it is bytes or MB or GB
that is measured. Also this is setting is exposed to users via --bandwidth on
command line.
My understanding about the nomenclature -- Not a native english language
speaker -- Bandwidth is the maximum channel capacity, throughput represents the
speed at which things move through a system. if my understanding is correct,
then the current naming seems to be more apt. I do see that we have used
bandwidthPerSec without units in earlier config properties. But I am conflicted
whether we should follow that pattern or use one which is more explicit.
if you feel strongly about this please file a JIRA and I will make a follow up
fix.
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch, HDFS-10551-HDFS-1312.003.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10563) Block reports could be silently dropped by NN
Staffan Friberg created HDFS-10563: -- Summary: Block reports could be silently dropped by NN Key: HDFS-10563 URL: https://issues.apache.org/jira/browse/HDFS-10563 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0-beta1 Reporter: Staffan Friberg Reading through the block reporting code I think I've spotted a case when block reports can silently be dropped and leave thread waiting indefinitely on a FutureTask that never will be executed. The BlockReportProcessingThread.enqueue method doesn't return any status on if the enqueuing of the task was successful and does not handle the case when the queue is full and offer return false. Going back through the call stack to BlockManager.runBlockOp, which indirectly calls enqueue with a FutureTask and then proceeds to do get() om the task. So if the internal queue in the BlockReportingProcessingThread is full, the BR would never be handled and the thread queuing the task would wait indefinitely on the FutureTask that will never be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344955#comment-15344955
]
Hadoop QA commented on HDFS-10551:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} HDFS-1312 passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
32s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c88012f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812570/HDFS-10551-HDFS-1312.003.patch
|
| JIRA Issue | HDFS-10551 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux d359316a56e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-1312 / 4f5beb6 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/15875/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/15875/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/15875/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/15875/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15875/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-10473) Allow only suitable storage policies to be set on striped files
[ https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344952#comment-15344952 ] Hudson commented on HDFS-10473: --- SUCCESS: Integrated in Hadoop-trunk-Commit #1 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/1/]) HDFS-10473: Allow only suitable storage policies to be set on striped (uma.gangumalla: rev 17eae9ebb30a3b106c4f6ae0c5374a3ab83abd8a) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java > Allow only suitable storage policies to be set on striped files > --- > > Key: HDFS-10473 > URL: https://issues.apache.org/jira/browse/HDFS-10473 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10473-01.patch, HDFS-10473-02.patch, > HDFS-10473-03.patch, HDFS-10473-04.patch, HDFS-10473-05.patch > > > Currently some of existing storage policies are not suitable for striped > layout files. > This JIRA proposes to reject setting storage policy on striped files. > Another thought is to allow only suitable storage polices like ALL_SSD. > Since the major use case of EC is for cold data, this may not be at high > importance. So, I am ok to reject setting storage policy on striped files at > this stage. Please suggest if others have some thoughts on this. > Thanks [~zhz] for offline discussion on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344941#comment-15344941
]
Mingliang Liu commented on HDFS-10551:
--
Thanks [~eddyxu]] for reporting this and [~anu] for the patch. The v2 patch
looks overall good to me. Some minor comments:
# In {{Command}} constructor, {{// These arguments are valid for all
commands.}} can be deleted.
# In {{TestDiskBalancerCommand}} other test cases can also use the class filed
{{conf}} (e.g. {{testReadClusterFromJson()}}). This will also address the
checkstyle warning I think.
# For the config key name {{dfs.disk.balancer.max.disk.throughputInMBperSec}},
I'd suggest {{dfs.disk.balancer.max.disk.bandwidthPerSec}}. I think
_throughput_ and _bandwidth_ have different meanings in different context and
bandwidth here is better. This also conforms to existing names like
{{dfs.image.transfer.bandwidthPerSec}}, {{dfs.balance.bandwidthPerSec}} and
{{dfs.datanode.balance.bandwidthPerSec}}, to name a few.
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch, HDFS-10551-HDFS-1312.003.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10555) Unable to loadFSEdits due to a failure in readCachePoolInfo
[
https://issues.apache.org/jira/browse/HDFS-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344938#comment-15344938
]
Jing Zhao commented on HDFS-10555:
--
+1. Thanks for the fix, Uma!
> Unable to loadFSEdits due to a failure in readCachePoolInfo
> ---
>
> Key: HDFS-10555
> URL: https://issues.apache.org/jira/browse/HDFS-10555
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Critical
> Attachments: HDFS-10555-00.patch
>
>
> Recently some tests are failing and unable to loadFSEdits due to a failure in
> readCachePoolInfo.
> Here in below code
> FSImageSerialization.java
> {code}
> }
> if ((flags & ~0x2F) != 0) {
> throw new IOException("Unknown flag in CachePoolInfo: " + flags);
> }
> {code}
> When all values of CachePool variable set to true, flags value & ~0x2F turns
> out to non zero value. So, this condition failing due to the addition of 0x20
> and changing value from ~0x1F to ~0x2F.
> May be to fix this issue, we may can change multiply value to ~0x3F
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344927#comment-15344927
]
Lei (Eddy) Xu commented on HDFS-10551:
--
+1 pending jenkins.
Thanks. It passes the tests locally on my laptop.
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch, HDFS-10551-HDFS-1312.003.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10473) Allow only suitable storage policies to be set on striped files
[ https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344931#comment-15344931 ] Uma Maheswara Rao G commented on HDFS-10473: Thank you [~jingzhao] and [~zhz] for reviews. I have just committed to trunk. I will look to update the documentation for this change. I will file a JIRA for that. Thanks > Allow only suitable storage policies to be set on striped files > --- > > Key: HDFS-10473 > URL: https://issues.apache.org/jira/browse/HDFS-10473 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10473-01.patch, HDFS-10473-02.patch, > HDFS-10473-03.patch, HDFS-10473-04.patch, HDFS-10473-05.patch > > > Currently some of existing storage policies are not suitable for striped > layout files. > This JIRA proposes to reject setting storage policy on striped files. > Another thought is to allow only suitable storage polices like ALL_SSD. > Since the major use case of EC is for cold data, this may not be at high > importance. So, I am ok to reject setting storage policy on striped files at > this stage. Please suggest if others have some thoughts on this. > Thanks [~zhz] for offline discussion on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-10551:
Attachment: HDFS-10551-HDFS-1312.003.patch
Cleaned up the checkstyle warning.
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch, HDFS-10551-HDFS-1312.003.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10562) DiskBalancer: update documentation on how to report issues and debug
Anu Engineer created HDFS-10562: --- Summary: DiskBalancer: update documentation on how to report issues and debug Key: HDFS-10562 URL: https://issues.apache.org/jira/browse/HDFS-10562 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer & mover Affects Versions: HDFS-1312 Reporter: Anu Engineer Assignee: Anu Engineer Priority: Minor Fix For: HDFS-1312 Add a section in the diskbalancer documentation on how to report issues and how to debug diskbalancer usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10473) Allow only suitable storage policies to be set on striped files
[ https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344890#comment-15344890 ] Jing Zhao commented on HDFS-10473: -- I can confirm the failed tests are unrelated. +1 on 05 patch. > Allow only suitable storage policies to be set on striped files > --- > > Key: HDFS-10473 > URL: https://issues.apache.org/jira/browse/HDFS-10473 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10473-01.patch, HDFS-10473-02.patch, > HDFS-10473-03.patch, HDFS-10473-04.patch, HDFS-10473-05.patch > > > Currently some of existing storage policies are not suitable for striped > layout files. > This JIRA proposes to reject setting storage policy on striped files. > Another thought is to allow only suitable storage polices like ALL_SSD. > Since the major use case of EC is for cold data, this may not be at high > importance. So, I am ok to reject setting storage policy on striped files at > this stage. Please suggest if others have some thoughts on this. > Thanks [~zhz] for offline discussion on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10552) DiskBalancer "-query" results in NPE if no plan for the node
[
https://issues.apache.org/jira/browse/HDFS-10552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344886#comment-15344886
]
Anu Engineer commented on HDFS-10552:
-
[~eddyxu] Failed tests are passing in HDFS-10551. I will commit both and post a
master patch against trunk soon.
> DiskBalancer "-query" results in NPE if no plan for the node
>
>
> Key: HDFS-10552
> URL: https://issues.apache.org/jira/browse/HDFS-10552
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: HDFS-1312
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10552-HDFS-1312.001.patch,
> HDFS-10552-HDFS-1312.002.patch, HDFS-10552-HDFS-1312.003.patch,
> HDFS-10552-HDFS-1312.004.patch
>
>
> {code}
> 16/06/20 11:50:16 INFO command.Command: Executing "query plan" command.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$QueryPlanStatusResponseProto$Builder.setPlanID(ClientDatanodeProtocolProtos.java:12782)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.queryDiskBalancerPlan(ClientDatanodeProtocolServerSideTranslatorPB.java:340)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:17513)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10396) Using -diff option with DistCp may get "Comparison method violates its general contract" exception
[
https://issues.apache.org/jira/browse/HDFS-10396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344769#comment-15344769
]
Reza Farivar commented on HDFS-10396:
-
Yongjun, do you have any updates on this? We are experiencing the same issue in
a distcp job.
Or can you tell us what the cause of this problem was that you were trying to
fix?
Thanks
> Using -diff option with DistCp may get "Comparison method violates its
> general contract" exception
> --
>
> Key: HDFS-10396
> URL: https://issues.apache.org/jira/browse/HDFS-10396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> Using -diff option get the following exception due to a bug in the comparison
> operator:
> {code}
> 16/04/21 14:34:18 WARN tools.DistCp: Failed to use snapshot diff for distcp
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:868)
> at java.util.TimSort.mergeAt(TimSort.java:485)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
> at java.util.TimSort.sort(TimSort.java:223)
> at java.util.TimSort.sort(TimSort.java:173)
> at java.util.Arrays.sort(Arrays.java:659)
> at org.apache.hadoop.tools.DistCpSync.moveToTarget(DistCpSync.java:293)
> at org.apache.hadoop.tools.DistCpSync.syncDiff(DistCpSync.java:261)
> at org.apache.hadoop.tools.DistCpSync.sync(DistCpSync.java:131)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:163)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> 16/04/21 14:34:18 ERROR tools.DistCp: Exception encountered
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344764#comment-15344764
]
Xiaobing Zhou commented on HDFS-10551:
--
[~anu] thanks for the patch. +1 v002 patch. Would you please fix the check
style?
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10551) o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected.
[
https://issues.apache.org/jira/browse/HDFS-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344765#comment-15344765
]
Xiaobing Zhou commented on HDFS-10551:
--
[~anu] thanks for the patch. +1 v002 patch. Would you please fix the check
style?
> o.a.h.h.s.diskbalancer.command.Command does not actually verify options as
> expected.
>
>
> Key: HDFS-10551
> URL: https://issues.apache.org/jira/browse/HDFS-10551
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Attachments: HDFS-10551-HDFS-1312.001.patch,
> HDFS-10551-HDFS-1312.002.patch
>
>
> In {{diskbalancer.command.Command#verifyCommandOptions}}. The following code
> does not do what it expected to do:
> {code}
> if (!validArgs.containsKey(opt.getArgName())) {
> {code}
> opt.getArgName() always returns "arg" instead of i.e., {{report}} or {{uri}},
> which is the expected parameter to check.
> It should use {{opt.getLongOpt()}} to get the option names. It can pass on
> the branch because {{opt.getArgName()}} always returns {{"arg"}}, which is
> accidently in {{validArgs}}. However I don't think it is the intention for
> this function.
> Additionally, in the following code
> {code}
> validArguments.append("Valid arguments are : %n");
> {code}
> This {{%n}} is not used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-1312) Re-balance disks within a Datanode
[
https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344753#comment-15344753
]
Hadoop QA commented on HDFS-1312:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project: The patch generated 19 new
+ 1000 unchanged - 2 fixed = 1019 total (was 1002) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
14s{color} | {color:green} The patch generated 0 new + 75 unchanged - 1 fixed =
75 total (was 76) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e2f6409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812506/HDFS-1312.003.patch |
| JIRA Issue | HDFS-1312 |
|
[jira] [Commented] (HDFS-10561) test_native_mini_dfs fails by NoClassDefFoundError
[
https://issues.apache.org/jira/browse/HDFS-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344649#comment-15344649
]
Hadoop QA commented on HDFS-10561:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e2f6409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812527/HDFS-10561.01.patch |
| JIRA Issue | HDFS-10561 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 0f32a57340d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4ee3543 |
| Default Java | 1.8.0_91 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15874/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15874/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> test_native_mini_dfs fails by NoClassDefFoundError
> --
>
> Key: HDFS-10561
> URL: https://issues.apache.org/jira/browse/HDFS-10561
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: native, test
>Affects Versions: 2.9.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-10561.01.patch
>
>
> After HDFS-10256, test_libhdfs_threaded_hdfs_static,
> test_libhdfs_zerocopy_hdfs_static, and test_test_native_mini_dfs are failing
> by NoClassDefFoundError.
> {noformat}
> nmdCreate: Builder#build error:
> java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:205)
> at
>
[jira] [Commented] (HDFS-10561) test_native_mini_dfs fails by NoClassDefFoundError
[
https://issues.apache.org/jira/browse/HDFS-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344622#comment-15344622
]
Akira AJISAKA commented on HDFS-10561:
--
Hi [~vinayrpet], would you review my patch?
> test_native_mini_dfs fails by NoClassDefFoundError
> --
>
> Key: HDFS-10561
> URL: https://issues.apache.org/jira/browse/HDFS-10561
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: native, test
>Affects Versions: 2.9.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-10561.01.patch
>
>
> After HDFS-10256, test_libhdfs_threaded_hdfs_static,
> test_libhdfs_zerocopy_hdfs_static, and test_test_native_mini_dfs are failing
> by NoClassDefFoundError.
> {noformat}
> nmdCreate: Builder#build error:
> java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:205)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:150)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:159)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2745)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2735)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:800)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:490)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:449)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 8 more
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_native_mini_dfs.c:35
> (errno: 2): got NULL from cl
>
> Test time = 1.05 sec
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10561) test_native_mini_dfs fails by NoClassDefFoundError
[
https://issues.apache.org/jira/browse/HDFS-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-10561:
-
Affects Version/s: 2.9.0
Target Version/s: 2.9.0
Status: Patch Available (was: Open)
> test_native_mini_dfs fails by NoClassDefFoundError
> --
>
> Key: HDFS-10561
> URL: https://issues.apache.org/jira/browse/HDFS-10561
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: native, test
>Affects Versions: 2.9.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-10561.01.patch
>
>
> After HDFS-10256, test_libhdfs_threaded_hdfs_static,
> test_libhdfs_zerocopy_hdfs_static, and test_test_native_mini_dfs are failing
> by NoClassDefFoundError.
> {noformat}
> nmdCreate: Builder#build error:
> java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:205)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:150)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:159)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2745)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2735)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:800)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:490)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:449)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 8 more
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_native_mini_dfs.c:35
> (errno: 2): got NULL from cl
>
> Test time = 1.05 sec
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10561) test_native_mini_dfs fails by NoClassDefFoundError
[
https://issues.apache.org/jira/browse/HDFS-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-10561:
-
Attachment: HDFS-10561.01.patch
Attaching a patch to add dependency to JUnit. This will fix
NoClassDefFoundError.
> test_native_mini_dfs fails by NoClassDefFoundError
> --
>
> Key: HDFS-10561
> URL: https://issues.apache.org/jira/browse/HDFS-10561
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: native, test
>Affects Versions: 2.9.0
>Reporter: Akira AJISAKA
> Attachments: HDFS-10561.01.patch
>
>
> After HDFS-10256, test_libhdfs_threaded_hdfs_static,
> test_libhdfs_zerocopy_hdfs_static, and test_test_native_mini_dfs are failing
> by NoClassDefFoundError.
> {noformat}
> nmdCreate: Builder#build error:
> java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:205)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:150)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:159)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2745)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2735)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:800)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:490)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:449)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 8 more
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_native_mini_dfs.c:35
> (errno: 2): got NULL from cl
>
> Test time = 1.05 sec
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10561) test_native_mini_dfs fails by NoClassDefFoundError
[
https://issues.apache.org/jira/browse/HDFS-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reassigned HDFS-10561:
Assignee: Akira AJISAKA
> test_native_mini_dfs fails by NoClassDefFoundError
> --
>
> Key: HDFS-10561
> URL: https://issues.apache.org/jira/browse/HDFS-10561
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: native, test
>Affects Versions: 2.9.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-10561.01.patch
>
>
> After HDFS-10256, test_libhdfs_threaded_hdfs_static,
> test_libhdfs_zerocopy_hdfs_static, and test_test_native_mini_dfs are failing
> by NoClassDefFoundError.
> {noformat}
> nmdCreate: Builder#build error:
> java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:205)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:150)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:159)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2745)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2735)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:800)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:490)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:449)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 8 more
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_native_mini_dfs.c:35
> (errno: 2): got NULL from cl
>
> Test time = 1.05 sec
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-1312) Re-balance disks within a Datanode
[ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-1312: Attachment: HDFS-1312.003.patch > Re-balance disks within a Datanode > -- > > Key: HDFS-1312 > URL: https://issues.apache.org/jira/browse/HDFS-1312 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode >Reporter: Travis Crawford >Assignee: Anu Engineer > Attachments: Architecture_and_test_update.pdf, > Architecture_and_testplan.pdf, HDFS-1312.001.patch, HDFS-1312.002.patch, > HDFS-1312.003.patch, disk-balancer-proposal.pdf > > > Filing this issue in response to ``full disk woes`` on hdfs-user. > Datanodes fill their storage directories unevenly, leading to situations > where certain disks are full while others are significantly less used. Users > at many different sites have experienced this issue, and HDFS administrators > are taking steps like: > - Manually rebalancing blocks in storage directories > - Decomissioning nodes & later readding them > There's a tradeoff between making use of all available spindles, and filling > disks at the sameish rate. Possible solutions include: > - Weighting less-used disks heavier when placing new blocks on the datanode. > In write-heavy environments this will still make use of all spindles, > equalizing disk use over time. > - Rebalancing blocks locally. This would help equalize disk use as disks are > added/replaced in older cluster nodes. > Datanodes should actively manage their local disk so operator intervention is > not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10557) Fix handling of the -fs Generic option
[ https://issues.apache.org/jira/browse/HDFS-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10557: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Fixed. Thanks for the code review [~anu]. > Fix handling of the -fs Generic option > -- > > Key: HDFS-10557 > URL: https://issues.apache.org/jira/browse/HDFS-10557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: diskbalancer >Affects Versions: HDFS-1312 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10557-HDFS-1312.02.patch, > HDFS-10557-HDFS-1312.03.patch, HDFS-10557-HDFS-1312.04.patch > > > A recent change to DiskBalancer replaced the -uri option with -fs. However > -fs is a [generic > option|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options] > so it is consumed by the GenericOptionsParser. > We can update this option handling to make it similar to other hdfs commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10561) test_native_mini_dfs fails by NoClassDefFoundError
Akira AJISAKA created HDFS-10561:
Summary: test_native_mini_dfs fails by NoClassDefFoundError
Key: HDFS-10561
URL: https://issues.apache.org/jira/browse/HDFS-10561
Project: Hadoop HDFS
Issue Type: Bug
Components: native, test
Reporter: Akira AJISAKA
After HDFS-10256, test_libhdfs_threaded_hdfs_static,
test_libhdfs_zerocopy_hdfs_static, and test_test_native_mini_dfs are failing by
NoClassDefFoundError.
{noformat}
nmdCreate: Builder#build error:
java.lang.NoClassDefFoundError: org/junit/Assert
at
org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:205)
at
org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:150)
at
org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:159)
at
org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2745)
at
org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2735)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:800)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:490)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:449)
Caused by: java.lang.ClassNotFoundException: org.junit.Assert
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
TEST_ERROR: failed on
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_native_mini_dfs.c:35
(errno: 2): got NULL from cl
Test time = 1.05 sec
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10557) Fix handling of the -fs Generic option
[ https://issues.apache.org/jira/browse/HDFS-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10557: - Attachment: HDFS-10557-HDFS-1312.04.patch v4 patch fixes the checkstyle issue. I will commit it shortly since the delta from v4 is trivial. > Fix handling of the -fs Generic option > -- > > Key: HDFS-10557 > URL: https://issues.apache.org/jira/browse/HDFS-10557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: diskbalancer >Affects Versions: HDFS-1312 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10557-HDFS-1312.02.patch, > HDFS-10557-HDFS-1312.03.patch, HDFS-10557-HDFS-1312.04.patch > > > A recent change to DiskBalancer replaced the -uri option with -fs. However > -fs is a [generic > option|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options] > so it is consumed by the GenericOptionsParser. > We can update this option handling to make it similar to other hdfs commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344361#comment-15344361
]
Hadoop QA commented on HDFS-10560:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 6s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-hdfs-project: The patch generated 1 new + 5
unchanged - 0 fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 19s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c88012f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812464/HDFS-10560-HDFS-1312.002.patch
|
| JIRA Issue | HDFS-10560 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 43925d706c9a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-1312 / 62e4dcd |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-10342) BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt
[
https://issues.apache.org/jira/browse/HDFS-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344366#comment-15344366
]
Kuhu Shukla commented on HDFS-10342:
Test failures are seen in previous builds as well. I am now trying to find the
JIRA that may have caused this and will file related test failure JIRAs.
Failing test {{TestDFSUpgradeWithHA}} passes locally and is not related.
Requesting [~kihwal], [~daryn] for comments/review. Thanks a lot!
> BlockManager#createLocatedBlocks should not check corrupt replicas if none
> are corrupt
> --
>
> Key: HDFS-10342
> URL: https://issues.apache.org/jira/browse/HDFS-10342
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Kuhu Shukla
> Attachments: HDFS-10342.001.patch, HDFS-10342.002.patch
>
>
> {{corruptReplicas#isReplicaCorrupt(block, node)}} is called for every node
> while populating the machines array. There's no need to invoke the method if
> {{corruptReplicas#numCorruptReplicas(block)}} returned 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10460) Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block
[
https://issues.apache.org/jira/browse/HDFS-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344331#comment-15344331
]
Kai Zheng commented on HDFS-10460:
--
Thanks for the update, Rakesh. It looks much close now:
1. In StripedBlockChecksumReconstructor#reconstruct(), could we move the most
of the new codes to a new function like
{{checksumWithTargetOutput(targetBuffer)}}? Doing this would separate the
checksum logic from the reconstruction stuffs and when we add to handle more
failures than one, the new function can be reused.
2. Ref. the following test case: 1) The 1 requestedLen case is good to have,
might also have a larger one like 10? 2) Don't quite get why we have an array
for the only one filepath. 3) You have quite a few tests like this, guess you
could move the codes of preparing for the test file to a common place.
{code}
+ /**
+ * Test to verify that the checksum can be computed for a small file less
than
+ * bytesPerCRC size.
+ */
+ @Test(timeout = 9)
+ public void testStripedFileChecksumWithMissedDataBlocksRangeQuery16()
+ throws Exception {
+int fileLength = 100;
+byte[] fileData = StripedFileTestUtil.generateBytes(fileLength);
+String stripedFile3 = ecDir + "/stripedFileChecksum3";
+String[] filePaths = new String[] {stripedFile3};
+
+for (String filePath : filePaths) {
+ Path testPath = new Path(filePath);
+ DFSTestUtil.writeFile(fs, testPath, fileData);
+}
+testStripedFileChecksumWithMissedDataBlocksRangeQuery(stripedFile3, 1);
+ }
{code}
> Erasure Coding: Recompute block checksum for a particular range less than
> file size on the fly by reconstructing missed block
> -
>
> Key: HDFS-10460
> URL: https://issues.apache.org/jira/browse/HDFS-10460
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-10460-00.patch, HDFS-10460-01.patch,
> HDFS-10460-02.patch, HDFS-10460-03.patch
>
>
> This jira is HDFS-9833 follow-on task to address reconstructing block and
> then recalculating block checksum for a particular range query.
> For example,
> {code}
> // create a file 'stripedFile1' with fileSize = cellSize * numDataBlocks =
> 65536 * 6 = 393216
> FileChecksum stripedFileChecksum = getFileChecksum(stripedFile1, 10, true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10536) Standby NN can not trigger log roll after EditLogTailer thread failed 3 times in EditLogTailer.triggerActiveLogRoll method.
[ https://issues.apache.org/jira/browse/HDFS-10536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344270#comment-15344270 ] XingFeng Shen commented on HDFS-10536: -- Looks like build failed [here|https://builds.apache.org/job/PreCommit-HDFS-Build/15871/console] may be due to env issue... Can you help to trigger it again ? > Standby NN can not trigger log roll after EditLogTailer thread failed 3 times > in EditLogTailer.triggerActiveLogRoll method. > --- > > Key: HDFS-10536 > URL: https://issues.apache.org/jira/browse/HDFS-10536 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 3.0.0-alpha1 >Reporter: XingFeng Shen >Priority: Critical > Labels: patch > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-10536-02.patch, HDFS-10536.patch > > > When all NameNodes become standby, EditLogTailer will retry 3 times to > trigger log roll, then it will be failed and throw Exception "Cannot find any > valid remote NN to service request!". After one namenode become active, > standby NN still can not trigger log roll again because variable > "nnLoopCount" is still 3, it can not init to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10560:
-
Attachment: HDFS-10560-HDFS-1312.002.patch
The failed test was related. It seemed these code lead the method
{{NodePlan#parseJson}} threw Exception.
{code}
public class NodePlan {
@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS,
include = JsonTypeInfo.As.PROPERTY, property = "@class")
{code}
I'd like to restore the code in {{NodePlan}}, post a new patch.
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10560-HDFS-1312.001.patch,
> HDFS-10560-HDFS-1312.002.patch
>
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10536) Standby NN can not trigger log roll after EditLogTailer thread failed 3 times in EditLogTailer.triggerActiveLogRoll method.
[ https://issues.apache.org/jira/browse/HDFS-10536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344160#comment-15344160 ] XingFeng Shen commented on HDFS-10536: -- [~vinayrpet], I had update the patch, please help to check it . > Standby NN can not trigger log roll after EditLogTailer thread failed 3 times > in EditLogTailer.triggerActiveLogRoll method. > --- > > Key: HDFS-10536 > URL: https://issues.apache.org/jira/browse/HDFS-10536 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 3.0.0-alpha1 >Reporter: XingFeng Shen >Priority: Critical > Labels: patch > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-10536-02.patch, HDFS-10536.patch > > > When all NameNodes become standby, EditLogTailer will retry 3 times to > trigger log roll, then it will be failed and throw Exception "Cannot find any > valid remote NN to service request!". After one namenode become active, > standby NN still can not trigger log roll again because variable > "nnLoopCount" is still 3, it can not init to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344112#comment-15344112
]
Hadoop QA commented on HDFS-10560:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-hdfs-project: The patch generated 1 new + 4
unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 12s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 0s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.diskbalancer.TestPlanner |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancerWithMockMover |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c88012f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812437/HDFS-10560-HDFS-1312.001.patch
|
| JIRA Issue | HDFS-10560 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux acb83724b12d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HDFS-10536) Standby NN can not trigger log roll after EditLogTailer thread failed 3 times in EditLogTailer.triggerActiveLogRoll method.
[ https://issues.apache.org/jira/browse/HDFS-10536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XingFeng Shen updated HDFS-10536: - Labels: patch (was: ) Fix Version/s: 3.0.0-alpha1 Affects Version/s: 3.0.0-alpha1 Target Version/s: 3.0.0-alpha1 Status: Patch Available (was: Open) > Standby NN can not trigger log roll after EditLogTailer thread failed 3 times > in EditLogTailer.triggerActiveLogRoll method. > --- > > Key: HDFS-10536 > URL: https://issues.apache.org/jira/browse/HDFS-10536 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 3.0.0-alpha1 >Reporter: XingFeng Shen >Priority: Critical > Labels: patch > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-10536-02.patch, HDFS-10536.patch > > > When all NameNodes become standby, EditLogTailer will retry 3 times to > trigger log roll, then it will be failed and throw Exception "Cannot find any > valid remote NN to service request!". After one namenode become active, > standby NN still can not trigger log roll again because variable > "nnLoopCount" is still 3, it can not init to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10536) Standby NN can not trigger log roll after EditLogTailer thread failed 3 times in EditLogTailer.triggerActiveLogRoll method.
[ https://issues.apache.org/jira/browse/HDFS-10536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XingFeng Shen updated HDFS-10536: - Attachment: HDFS-10536-02.patch > Standby NN can not trigger log roll after EditLogTailer thread failed 3 times > in EditLogTailer.triggerActiveLogRoll method. > --- > > Key: HDFS-10536 > URL: https://issues.apache.org/jira/browse/HDFS-10536 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Reporter: XingFeng Shen >Priority: Critical > Attachments: HDFS-10536-02.patch, HDFS-10536.patch > > > When all NameNodes become standby, EditLogTailer will retry 3 times to > trigger log roll, then it will be failed and throw Exception "Cannot find any > valid remote NN to service request!". After one namenode become active, > standby NN still can not trigger log roll again because variable > "nnLoopCount" is still 3, it can not init to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343971#comment-15343971
]
Yiqun Lin commented on HDFS-10560:
--
Thanks [~ajisakaa] for comment, rename the patch right now.
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10560-HDFS-1312.001.patch
>
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10560:
-
Attachment: (was: HDFS-10560.001.patch)
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10560-HDFS-1312.001.patch
>
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10560:
-
Attachment: HDFS-10560-HDFS-1312.001.patch
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10560-HDFS-1312.001.patch
>
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10557) Fix handling of the -fs Generic option
[
https://issues.apache.org/jira/browse/HDFS-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343964#comment-15343964
]
Hadoop QA commented on HDFS-10557:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 1s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c88012f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812418/HDFS-10557-HDFS-1312.03.patch
|
| JIRA Issue | HDFS-10557 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a36101a7c3fb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-1312 / 62e4dcd |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15867/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15867/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15867/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15867/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343960#comment-15343960
]
Akira AJISAKA commented on HDFS-10560:
--
Hi [~linyiqun], would you rename the patch to include the target branch name to
run precommit job on branch HDFS-1312?
https://yetus.apache.org/documentation/in-progress/precommit-patchnames/
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10560.001.patch
>
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343945#comment-15343945
]
Hadoop QA commented on HDFS-10560:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HDFS-10560 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812432/HDFS-10560.001.patch |
| JIRA Issue | HDFS-10560 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15869/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10560.001.patch
>
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10560:
-
Attachment: HDFS-10560.001.patch
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10560.001.patch
>
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10560:
-
Status: Patch Available (was: Open)
This new feature of {{DiskBalancer}} seems pretty good. This is my first patch
for this feature. I am looking forward to seeing the reply.
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10560) DiskBalancer: Reuse ObjectMapper instance to improve the performance
[
https://issues.apache.org/jira/browse/HDFS-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10560:
-
Summary: DiskBalancer: Reuse ObjectMapper instance to improve the
performance (was: Reuse ObjectMapper instance to improve the performance)
> DiskBalancer: Reuse ObjectMapper instance to improve the performance
>
>
> Key: HDFS-10560
> URL: https://issues.apache.org/jira/browse/HDFS-10560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>
> In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to
> do the json-object transform. But {{ObjectMapper}} instance is relatively
> heavy, we should reuse them as possible. And In addition, {{ObjectMapper}} is
> thread safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
> Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10560) Reuse ObjectMapper instance to improve the performance
Yiqun Lin created HDFS-10560:
Summary: Reuse ObjectMapper instance to improve the performance
Key: HDFS-10560
URL: https://issues.apache.org/jira/browse/HDFS-10560
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer & mover
Affects Versions: HDFS-1312
Reporter: Yiqun Lin
Assignee: Yiqun Lin
In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to do
the json-object transform. But {{ObjectMapper}} instance is relatively heavy,
we should reuse them as possible. And In addition, {{ObjectMapper}} is thread
safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10557) Fix handling of the -fs Generic option
[
https://issues.apache.org/jira/browse/HDFS-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343914#comment-15343914
]
Hadoop QA commented on HDFS-10557:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 14s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c88012f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12812406/HDFS-10557-HDFS-1312.02.patch
|
| JIRA Issue | HDFS-10557 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 318f44aa7cb6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-1312 / 62e4dcd |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15866/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15866/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15866/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Updated] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10556:
-
Status: Patch Available (was: Open)
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10556-branch-2.000.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10556:
-
Attachment: HDFS-10556-branch-2.000.patch
The v0 patch simply calls the {{validate()}} private method in setters of the
dependent (or exclusive) options. This is very similar to the code before
[HDFS-10397]. It also adds a new unit test to test constructing
{{DistCpOptions}} manually with setters. Its test cases are very similar to the
parser test in {{TestOptionsParser}}, except those who are validated in parser
instead of the {{DistCpOptions#validate()}}.
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10556-branch-2.000.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10556) DistCpOptions should be validated automatically
[
https://issues.apache.org/jira/browse/HDFS-10556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343858#comment-15343858
]
Mingliang Liu edited comment on HDFS-10556 at 6/22/16 7:47 AM:
---
The v0 patch simply calls the {{validate()}} private method in setters of the
dependent (or exclusive) options. The basic idea is kinda similar to the code
before [HDFS-10397]. It also adds a new unit test to test constructing
{{DistCpOptions}} manually with setters. Its test cases are very similar to the
parser test in {{TestOptionsParser}}, except those who are validated in parser
instead of the {{DistCpOptions#validate()}}.
was (Author: liuml07):
The v0 patch simply calls the {{validate()}} private method in setters of the
dependent (or exclusive) options. This is very similar to the code before
[HDFS-10397]. It also adds a new unit test to test constructing
{{DistCpOptions}} manually with setters. Its test cases are very similar to the
parser test in {{TestOptionsParser}}, except those who are validated in parser
instead of the {{DistCpOptions#validate()}}.
> DistCpOptions should be validated automatically
> ---
>
> Key: HDFS-10556
> URL: https://issues.apache.org/jira/browse/HDFS-10556
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10556-branch-2.000.patch
>
>
> {{DistCpOptions}} can be set from command-line or may be set manually. In
> [HDFS-10397], we refactored the validation to make it simpler and more
> efficient. However, the newly added {{validate()}} method may not be
> automatically revoked. This is the major concern for existing downstreams
> that create the {{DistCpOptions}} manually instead of by parser, and have
> conflicting options.
> This jira is to make the validation happen automatically. A simple fix is to
> use the approach that validates in individual setters. This is a fix for
> {{branch-2}}. As a long-term fix, in [HDFS-10533], we're making the
> {{DistCpOptions}} immutable so that it will be hard, if not impossible, to
> use it wrongly by downstream applications. However, that code will only go to
> {{trunk}} branch as it breaks backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10557) Fix handling of the -fs Generic option
[ https://issues.apache.org/jira/browse/HDFS-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10557: - Attachment: (was: HDFS-10557-HDFS-1312.03.patch) > Fix handling of the -fs Generic option > -- > > Key: HDFS-10557 > URL: https://issues.apache.org/jira/browse/HDFS-10557 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: diskbalancer >Affects Versions: HDFS-1312 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10557-HDFS-1312.02.patch, > HDFS-10557-HDFS-1312.03.patch > > > A recent change to DiskBalancer replaced the -uri option with -fs. However > -fs is a [generic > option|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options] > so it is consumed by the GenericOptionsParser. > We can update this option handling to make it similar to other hdfs commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
