[jira] [Created] (HDFS-13236) Standby NN down with error encountered while tailing edits

[Yuriy Malygin (JIRA)]

Yuriy Malygin created HDFS-13236:


 Summary: Standby NN down with error encountered while tailing edits
 Key: HDFS-13236
 URL: https://issues.apache.org/jira/browse/HDFS-13236
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: journal-node, namenode
Affects Versions: 3.0.0
Reporter: Yuriy Malygin


After update Hadoop from 2.7.3 to 3.0.0 standby NN down with error encountered 
while tailing edits from JN:
{code:java}
Feb 28 01:58:31 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:31,594 INFO 
[FSImageSaver for /one/hadoop-data/dfs of type IMAGE_AND_EDITS] 
FSImageFormatProtobuf - Image file 
/one/hadoop-data/dfs/current/fsimage.ckpt_012748979
98 of size 4595971949 bytes saved in 93 seconds.
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Going to retain 2 
images with txid >= 1274897935
Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445 INFO 
[Standby State Checkpointer] NNStorageRetentionManager - Purging old image 
FSImageFile(file=/one/hadoop-data/dfs/current/fsimage_01274897875, 
cpktTxId
=01274897875)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Reading 
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@6a168e6f 
expecting start txid #1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660 INFO 
[Edit log tailer] FSImage - Start loading edits file 
http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A10
56233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true, 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848;
inProgressOk=true
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999
torageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true,
 
http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217
-aef5-6ed206893848=true' to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661 INFO 
[Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream 
'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup=1274897999=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848=true'
 to transaction ID 1274897999
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,680 ERROR 
[Edit log tailer] FSEditLogLoader - Encountered exception on operation AddOp 
[length=0, inodeId=145550319, 
path=/kafka/parquet/infrastructureGrace/date=2018-02-28/_temporary/1/_temporary/attempt_1516181147167_20856_r_98_0/part-r-00098.gz.parquet,
 replication=3, mtime=1519772206615, atime=1519772206615, blockSize=134217728, 
blocks=[], permissions=root:supergroup:rw-r--r--, aclEntries=null, 
clientName=DFSClient_attempt_1516181147167_20856_r_98_0_1523538799_1, 
clientMachine=10.137.2.142, overwrite=false, RpcClientId=, RpcCallId=271996603, 
storagePolicyId=0, erasureCodingPolicyId=0, opCode=OP_ADD, txid=1274898002]
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 
java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected 
length 16
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:946)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
Feb 28 01:58:34 

[jira] [Created] (HDFS-13235) DiskBalancer: Update Documentation to add newly added options

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created HDFS-13235:
-

 Summary: DiskBalancer: Update Documentation to add newly added 
options
 Key: HDFS-13235
 URL: https://issues.apache.org/jira/browse/HDFS-13235
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13234) Remove renew configuration instance in ConfiguredFailoverProxyProvider and reduce memory footprint for client

[He Xiaoqiao (JIRA)]

He Xiaoqiao created HDFS-13234:
--

 Summary: Remove renew configuration instance in 
ConfiguredFailoverProxyProvider and reduce memory footprint for client
 Key: HDFS-13234
 URL: https://issues.apache.org/jira/browse/HDFS-13234
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: fs, ha, hdfs-client
Reporter: He Xiaoqiao


The memory footprint of #DFSClient is very considerable in some special 
scenario since there are many #Configuration instances and occupy much memory 
resource (In an extreme case, org.apache.hadoop.conf.Configuration occupies 
over 600MB we meet under HDFS Federation an HA with QJM and there are dozens of 
NameNodes). I think some new Configuration instance is not necessary. Such as  
#ConfiguredFailoverProxyProvider initialization.

{code:java}
  public ConfiguredFailoverProxyProvider(Configuration conf, URI uri,
  Class xface, HAProxyFactory factory) {
this.xface = xface;
this.conf = new Configuration(conf);
..
  }
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13233) RBF：getMountPoint return wrong mount point if the input path contains the mount point

[wangzhiyuan (JIRA)]

wangzhiyuan created HDFS-13233:
--

 Summary: RBF：getMountPoint return wrong mount point if the input 
path contains the mount point
 Key: HDFS-13233
 URL: https://issues.apache.org/jira/browse/HDFS-13233
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 3.2.0
Reporter: wangzhiyuan


Suppose the mount point table include: "/" "/user"  "/user/test" "/user/test1", 
if the input path is "/user/test111", the return mount point of 
(MountTableResolver->getMountPoint) is "/user/test1", but the correct one 
should be "/user". 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Valid path clarification for Hadoop

[Zsolt Venczel]

I think it's a great peace of information to start with. Thanks!

On Mon, Mar 5, 2018 at 5:13 PM, Anu Engineer 
wrote:

> Not exactly what you want, but here are the docs from the user perspective.
>
> http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/
> hadoop-common/filesystem/introduction.html#Path_Names
>
> --Anu
>
>
> On 3/5/18, 5:08 PM, "Zsolt Venczel"  wrote:
>
> Hi Hdfs Devs,
>
> While focusing on https://issues.apache.org/jira/browse/HDFS-13176 I
> was
> assembling a set of characters to test webhdfs file path with and was
> unable to find such an "official" list.
> Would it make sens to come up with a definition for supported path
> patterns
> for Hdfs? If there is such a definition can you please guide me to it?
>
> Thanks and best regards,
> Zsolt
>
>
>

Re: Valid path clarification for Hadoop

[Anu Engineer]

Not exactly what you want, but here are the docs from the user perspective.

http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/filesystem/introduction.html#Path_Names

--Anu


On 3/5/18, 5:08 PM, "Zsolt Venczel"  wrote:

Hi Hdfs Devs,

While focusing on https://issues.apache.org/jira/browse/HDFS-13176 I was
assembling a set of characters to test webhdfs file path with and was
unable to find such an "official" list.
Would it make sens to come up with a definition for supported path patterns
for Hdfs? If there is such a definition can you please guide me to it?

Thanks and best regards,
Zsolt

Valid path clarification for Hadoop

[Zsolt Venczel]

Hi Hdfs Devs,

While focusing on https://issues.apache.org/jira/browse/HDFS-13176 I was
assembling a set of characters to test webhdfs file path with and was
unable to find such an "official" list.
Would it make sens to come up with a definition for supported path patterns
for Hdfs? If there is such a definition can you please guide me to it?

Thanks and best regards,
Zsolt

[jira] [Created] (HDFS-13232) RBF: ConnectionPool should return first usable connection

[Wei Yan (JIRA)]

Wei Yan created HDFS-13232:
--

 Summary: RBF: ConnectionPool should return first usable connection
 Key: HDFS-13232
 URL: https://issues.apache.org/jira/browse/HDFS-13232
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Wei Yan
Assignee: Wei Yan


In current ConnectionPool.getConnection(), it will return the first active 
connection:
{code:java}
for (int i=0; i

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Andrew Wang]

Hi Sanjay, thanks for the response, replying inline:

- NN on top HDSL where the NN uses the new block layer (Both Daryn and Owen
> acknowledge the benefit of the new block layer).  We have two choices here
>  ** a) Evolve NN so that it can interact with both old and new block layer,
>  **  b) Fork and create new NN that works only with new block layer, the
> old NN will continue to work with old block layer.
> There are trade-offs but clearly the 2nd option has least impact on the
> old HDFS code.
>
> Are you proposing that we pursue the 2nd option to integrate HDSL with
HDFS?


> - Share the HDSL’s netty  protocol engine with HDFS block layer.  After
> HDSL and Ozone has stabilized the engine, put the new netty engine in
> either HDFS or in Hadoop common - HDSL will use it from there. The HDFS
> community  has been talking about moving to better thread model for HDFS
> DNs since release 0.16!!
>
> The Netty-based protocol engine seems like it could be contributed
separately from HDSL. I'd be interested to learn more about the performance
and other improvements from this new engine.


> - Shallow copy. Here HDSL needs a way to get the actual linux file system
> links - HDFS block layer needs  to provide a private secure API to get file
> names of blocks so that HDSL can do a hard link (hence shallow copy)o
>

Why isn't this possible with two processes? SCR for instance securely
passes file descriptors between the DN and client over a unix domain
socket. I'm sure we can construct a protocol that securely and efficiently
creates hardlinks.

It also sounds like this shallow copy won't work with features like HDFS
encryption or erasure coding, which diminishes its utility. We also don't
even have HDFS-to-HDFS shallow copy yet, so HDFS-to-Ozone shallow copy is
even further out.

Best,
Andrew

[EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

[Cross-posting, as this affects the rest of the project]

Hey folks-

As discussed last month [1], the HDFS build hasn't been healthy
recently. We're dedicating a bug bash to stabilize the build and
address some longstanding issues with our unit tests. We rely on our
CI infrastructure to keep the project releasable, and in its current
state, it's not protecting us from regressions. While we probably
won't achieve all our goals in this session, we can develop the
conditions for reestablishing a firm foundation.

If you're new to the project, please consider attending and
contributing. Committers often prioritize large or complicated
patches, and the issues that make the project livable don't get enough
attention. A bug bash is a great opportunity to pull reviewers'
collars, and fix the annoyances that slow us all down.

If you're a committer, please join us! While some of the proposed
repairs are rote, many unit tests rely on implementation details and
non-obvious invariants. We need domain experts to help untangle
complex dependencies and to prevent breakage of deliberate, but
counter-intuitive code.

We're collecting tasks in wiki [2] and will include a dial-in option
for folks who aren't local.

Meetup has started charging for creating new events, so we'll have to
find another way to get an approximate headcount and publish the
address. Please ping me if you have a preferred alternative. -C

[1]: https://s.apache.org/nEoQ
[2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Andrew Wang]

Hi Owen, Wangda,

Thanks for clearly laying out the subproject options, that helps the
discussion.

I'm all onboard with the idea of regular releases, and it's something I
tried to do with the 3.0 alphas and betas. The problem though isn't a lack
of commitment from feature developers like Sanjay or Jitendra; far from it!
I think every feature developer makes a reasonable effort to test their
code before it's merged. Yet, my experience as an RM is that more code
comes with more risk. I don't believe that Ozone is special or different in
this regard. It comes with a maintenance cost, not a maintenance benefit.

I'm advocating for #3: separate source, separate release. Since HDSL
stability and FSN/BM refactoring are still a ways out, I don't want to
incur a maintenance cost now. I sympathize with the sentiment that working
cross-repo is harder than within same repo, but the right tooling can make
this a lot easier (e.g. git submodule, Google's repo tool). We have
experience doing this internally here at Cloudera, and I'm happy to share
knowledge and possibly code.

Best,
Andrew

On Fri, Mar 2, 2018 at 4:41 PM, Wangda Tan  wrote:

> I like the idea of same source / same release and put Ozone's source under
> a different directory.
>
> Like Owen mentioned, It gonna be important for all parties to keep a
> regular and shorter release cycle for Hadoop, e.g. 3-4 months between minor
> releases. Users can try features and give feedbacks to stabilize feature
> earlier; developers can be happier since efforts will be consumed by users
> soon after features get merged. In addition to this, if features merged to
> trunk after reasonable tests/review, Andrew's concern may not be a problem
> anymore:
>
> bq. Finally, I earnestly believe that Ozone/HDSL itself would benefit from
> being a separate project. Ozone could release faster and iterate more
> quickly if it wasn't hampered by Hadoop's release schedule and security and
> compatibility requirements.
>
> Thanks,
> Wangda
>
>
> On Fri, Mar 2, 2018 at 4:24 PM, Owen O'Malley 
> wrote:
>
>> On Thu, Mar 1, 2018 at 11:03 PM, Andrew Wang 
>> wrote:
>>
>> Owen mentioned making a Hadoop subproject; we'd have to
>> > hash out what exactly this means (I assume a separate repo still
>> managed by
>> > the Hadoop project), but I think we could make this work if it's more
>> > attractive than incubation or a new TLP.
>>
>>
>> Ok, there are multiple levels of sub-projects that all make sense:
>>
>>- Same source tree, same releases - examples like HDFS & YARN
>>- Same master branch, separate releases and release branches - Hive's
>>Storage API vs Hive. It is in the source tree for the master branch,
>> but
>>has distinct releases and release branches.
>>- Separate source, separate release - Apache Commons.
>>
>> There are advantages and disadvantages to each. I'd propose that we use
>> the
>> same source, same release pattern for Ozone. Note that we tried and later
>> reverted doing Common, HDFS, and YARN as separate source, separate release
>> because it was too much trouble. I like Daryn's idea of putting it as a
>> top
>> level directory in Hadoop and making sure that nothing in Common, HDFS, or
>> YARN depend on it. That way if a Release Manager doesn't think it is ready
>> for release, it can be trivially removed before the release.
>>
>> One thing about using the same releases, Sanjay and Jitendra are signing
>> up
>> to make much more regular bugfix and minor releases in the near future.
>> For
>> example, they'll need to make 3.2 relatively soon to get it released and
>> then 3.3 somewhere in the next 3 to 6 months. That would be good for the
>> project. Hadoop needs more regular releases and fewer big bang releases.
>>
>> .. Owen
>>
>
>

[jira] [Created] (HDFS-13231) Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI

[Haibo Yan (JIRA)]

Haibo Yan created HDFS-13231:


 Summary: Extend visualization for Maintenance Mode under Datanode 
tab in the NameNode UI
 Key: HDFS-13231
 URL: https://issues.apache.org/jira/browse/HDFS-13231
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, namenode
Affects Versions: 3.0.1
Reporter: Haibo Yan


With HDFS-9391, table view is using css dynamic class name to match the state

{code:html|title=hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html}
{name} ({xferaddr})
{code}

Some css is missing when the datanode is going to 


{code:javascript|title=hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js}
  if (n.adminState === "In Service") {
n.state = "alive";
  } else if (nodes[i].adminState === "Decommission In Progress") {
n.state = "decommissioning";
  } else if (nodes[i].adminState === "Decommissioned") {
n.state = "decommissioned";
  } else if (nodes[i].adminState === "Entering Maintenance") {
n.state = "entering-maintenance";
  } else if (nodes[i].adminState === "In Maintenance") {
n.state = "in-maintenance";
  }
{code}

dfshealth-node-decommissioning, dfshealth-node-entering-maintenance, 
dfshealth-node-in-maintenance should be added into hadoop.css





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-10992) file is under construction but no leases found

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-10992.

Resolution: Duplicate

> file is under construction but no leases found
> --
>
> Key: HDFS-10992
> URL: https://issues.apache.org/jira/browse/HDFS-10992
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.1
> Environment: hortonworks 2.3 build 2557. 10 Datanodes , 2 NameNode in 
> auto failover
>Reporter: Chernishev Aleksandr
>Priority: Major
>
> On hdfs after recording a small number of files (at least 1000) the size 
> (150Mb - 1,6Gb) found 13 damaged files with incomplete last block.
> hadoop fsck /hadoop/files/load_tarifer-zf-4_20160902165521521.csv 
> -openforwrite -files -blocks -locations
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
> Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8
> Connecting to namenode via 
> http://hadoop-hdfs:50070/fsck?ugi=hdfs=1=1=1=1=%2Fstaging%2Flanding%2Fstream%2Fitc_dwh%2Ffiles%2Fload_tarifer-zf-4_20160902165521521.csv
> FSCK started by hdfs (auth:SIMPLE) from /10.0.0.178 for path 
> /hadoop/files/load_tarifer-zf-4_20160902165521521.csv at Mon Oct 10 17:12:25 
> MSK 2016
> /hadoop/files/load_tarifer-zf-4_20160902165521521.csv 920596121 bytes, 7 
> block(s), OPENFORWRITE:  MISSING 1 blocks of total size 115289753 B
> 0. BP-1552885336-10.0.0.178-1446159880991:blk_1084952841_17798971 
> len=134217728 repl=4 
> [DatanodeInfoWithStorage[10.0.0.188:50010,DS-9ba44a76-113a-43ac-87dc-46aa97ba3267,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.183:50010,DS-eccd375a-ea32-491b-a4a3-5ea3faca4171,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.184:50010,DS-ec462491-6766-490a-a92f-38e9bb3be5ce,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.182:50010,DS-cef46399-bb70-4f1a-ac55-d71c7e820c29,DISK]]
> 1. BP-1552885336-10.0.0.178-1446159880991:blk_1084952850_17799207 
> len=134217728 repl=3 
> [DatanodeInfoWithStorage[10.0.0.184:50010,DS-412769e0-0ec2-48d3-b644-b08a516b1c2c,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.181:50010,DS-97388b2f-c542-417d-ab06-c8d81b94fa9d,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.187:50010,DS-e7a11951-4315-4425-a88b-a9f6429cc058,DISK]]
> 2. BP-1552885336-10.0.0.178-1446159880991:blk_1084952857_17799489 
> len=134217728 repl=3 
> [DatanodeInfoWithStorage[10.0.0.184:50010,DS-7a08c597-b0f4-46eb-9916-f028efac66d7,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.180:50010,DS-fa6a4630-1626-43d8-9988-955a86ac3736,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.182:50010,DS-8670e77d-c4db-4323-bb01-e0e64bd5b78e,DISK]]
> 3. BP-1552885336-10.0.0.178-1446159880991:blk_1084952866_17799725 
> len=134217728 repl=3 
> [DatanodeInfoWithStorage[10.0.0.185:50010,DS-b5ff8ba0-275e-4846-b5a4-deda35aa0ad8,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.180:50010,DS-9cb6cade-9395-4f3a-ab7b-7fabd400b7f2,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.183:50010,DS-e277dcf3-1bce-4efd-a668-cd6fb2e10588,DISK]]
> 4. BP-1552885336-10.0.0.178-1446159880991:blk_1084952872_17799891 
> len=134217728 repl=4 
> [DatanodeInfoWithStorage[10.0.0.184:50010,DS-e1d8f278-1a22-4294-ac7e-e12d554aef7f,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.186:50010,DS-5d9aeb2b-e677-41cd-844e-4b36b3c84092,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.183:50010,DS-eccd375a-ea32-491b-a4a3-5ea3faca4171,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.182:50010,DS-8670e77d-c4db-4323-bb01-e0e64bd5b78e,DISK]]
> 5. BP-1552885336-10.0.0.78-1446159880991:blk_1084952880_17800120 
> len=134217728 repl=3 
> [DatanodeInfoWithStorage[10.0.0.181:50010,DS-79185b75-1938-4c91-a6d0-bb6687ca7e56,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.184:50010,DS-dcbd20aa-0334-49e0-b807-d2489f5923c6,DISK],
>  
> DatanodeInfoWithStorage[10.0.0.183:50010,DS-f1d77328-f3af-483e-82e9-66ab0723a52c,DISK]]
> 6. 
> BP-1552885336-10.0.0.178-1446159880991:blk_1084952887_17800316{UCState=COMMITTED,
>  truncateBlock=null, primaryNodeIndex=-1, 
> replicas=[ReplicaUC[[DISK]DS-5f3eac72-eb55-4df7-bcaa-a6fa35c166a0:NORMAL:10.0.0.188:50010|RBW],
>  
> ReplicaUC[[DISK]DS-a2a0d8f0-772e-419f-b4ff-10b4966c57ca:NORMAL:10.0.0.184:50010|RBW],
>  
> ReplicaUC[[DISK]DS-52984aa0-598e-4fff-acfa-8904ca7b585c:NORMAL:10.0.0.185:50010|RBW]]}
>  len=115289753 MISSING!
> Status: CORRUPT
>  Total size:  920596121 B
>  Total dirs:  0
>  Total files: 1
>  Total symlinks:  0
>  Total blocks (validated):7 (avg. block size 131513731 B)
>   
>   UNDER MIN REPL'D BLOCKS:1 (14.285714 %)
>   dfs.namenode.replication.min:   1
>   CORRUPT FILES:  1
>   MISSING BLOCKS: 1
>   MISSING SIZE:   115289753 B
>   
>  Minimally replicated blocks: 6 (85.71429 %)
>  Over-replicated blocks:  2 

[jira] [Created] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns

[Wei Yan (JIRA)]

Wei Yan created HDFS-13230:
--

 Summary: RBF: ConnectionManager's cleanup task will compare each 
pool's own active conns with its total conns
 Key: HDFS-13230
 URL: https://issues.apache.org/jira/browse/HDFS-13230
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Wei Yan


In the cleanup task:
{code:java}
long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
int total = pool.getNumConnections();
int active = getNumActiveConnections();
if (timeSinceLastActive > connectionCleanupPeriodMs ||
{code}
the 3rd line should be pool.getNumActiveConnections()

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13229) Ozone: Add support for rename key within a bucket for rest client

[Lokesh Jain (JIRA)]

Lokesh Jain created HDFS-13229:
--

 Summary: Ozone: Add support for rename key within a bucket for 
rest client
 Key: HDFS-13229
 URL: https://issues.apache.org/jira/browse/HDFS-13229
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Lokesh Jain
Assignee: Lokesh Jain


This jira aims to add support for rename key within a bucket for rest client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13228) Ozone: Add support for rename key within a bucket for rpc client

[Lokesh Jain (JIRA)]

Lokesh Jain created HDFS-13228:
--

 Summary: Ozone: Add support for rename key within a bucket for rpc 
client
 Key: HDFS-13228
 URL: https://issues.apache.org/jira/browse/HDFS-13228
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Lokesh Jain
Assignee: Lokesh Jain


This jira aims to implement rename operation on a key within a bucket for rpc 
client. OzoneFilesystem currently rewrites a key on rename. Addition of this 
operation would simplify renames in OzoneFilesystem as renames would just be a 
db update in ksm.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/

[Mar 4, 2018 3:12:52 PM] (aajisaka) HADOOP-15286. Remove unused imports from 
TestKMSWithZK.java
[Mar 4, 2018 3:33:47 PM] (aajisaka) HADOOP-15282. HADOOP-15235 broke 
TestHttpFSServerWebServer




-1 overall


The following subsystems voted -1:
findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/diff-compile-javac-root.txt
  [296K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/whitespace-eol.txt
  [9.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/whitespace-tabs.txt
  [288K]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [48K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/712/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13227) Add a method to calculate cumulative diff over multiple snapshots in DirectoryDiffList

[Shashikant Banerjee (JIRA)]

Shashikant Banerjee created HDFS-13227:
--

 Summary: Add a method to  calculate cumulative diff over multiple 
snapshots in DirectoryDiffList
 Key: HDFS-13227
 URL: https://issues.apache.org/jira/browse/HDFS-13227
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: snapshots
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee


This Jira proposes to add an API in DirectoryWithSnapshotFeature#f 
DirectoryDiffList which will return minimal list of diffs needed to combine to 
get the cumulative diff between two given snapshots. The same method will be 
made use while constructing the childrenList for a directory. 
DirectoryWithSnapshotFeature#getChildrenList     and 
DirectoryWithSnapshotFeature#computeDiffBetweenSnapshots will make use of the 
same method to get the cumulative diff. When snapshotSkipList, with minimal set 
of diffs to combine in order to get the cumulative diff, the overall 
computation will be faster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry

[maobaolong (JIRA)]

maobaolong created HDFS-13226:
-

 Summary: RBF: We should throw the failure validate and refuse this 
mount entry
 Key: HDFS-13226
 URL: https://issues.apache.org/jira/browse/HDFS-13226
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs
Affects Versions: 3.2.0
Reporter: maobaolong


one of the mount entry source path rule is that the source path must start with 
'\', somebody didn't follow the rule and execute the following command:

{code:bash}
$ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
{code}

But, the console show we are successful add this entry.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-7240 to trunk

[sanjay Radia]

Andrew
  Thanks for your response. 

 In this email let me focus on maintenance and unnecessary impact on HDFS.
Daryn also touched on this topic and looked at the code base from the developer 
impact point of view. He appreciated that the code is separate and I agree with 
his suggestion to move it further up the src tree (e.g. Hadoop-hdsl-project or 
hadoop-hdfs-project/hadoop-hdsl). He also gave a good analogy to the store: do 
not break things as you change and evolve the store. Let’s look at the areas of 
future interaction as examples.

- NN on top HDSL where the NN uses the new block layer (Both Daryn and Owen 
acknowledge the benefit of the new block layer).  We have two choices here 
 ** a) Evolve NN so that it can interact with both old and new block layer, 
 **  b) Fork and create new NN that works only with new block layer, the old NN 
will continue to work with old block layer. 
There are trade-offs but clearly the 2nd option has least impact on the old 
HDFS code.  

- Share the HDSL’s netty  protocol engine with HDFS block layer.  After HDSL 
and Ozone has stabilized the engine, put the new netty engine in either HDFS or 
in Hadoop common - HDSL will use it from there. The HDFS community  has been 
talking about moving to better thread model for HDFS DNs since release 0.16!!

- Shallow copy. Here HDSL needs a way to get the actual linux file system links 
- HDFS block layer needs  to provide a private secure API to get file names of 
blocks so that HDSL can do a hard link (hence shallow copy)o

The first 2 examples are beneficial to existing HDFS and the maintenance burden 
can be minimized and worth the benefits (2x NN scalability!! And more efficient 
protocol engine). The 3rd is only beneficial to HDFS users who want the 
scalability of the new HDSL/Ozone code in a side-by-side system; here the cost 
is providing a  private API to access the block file name. 


sanjay

> On Mar 1, 2018, at 11:03 PM, Andrew Wang  wrote:
> 
> Hi Sanjay,
> 
> I have different opinions about what's important and how to eventually
> integrate this code, and that's not because I'm "conveniently ignoring"
> your responses. I'm also not making some of the arguments you claim I am
> making. Attacking arguments I'm not making is not going to change my mind,
> so let's bring it back to the arguments I am making.
> 
> Here's what it comes down to: HDFS-on-HDSL is not going to be ready in the
> near-term, and it comes with a maintenance cost.
> 
> I did read the proposal on HDFS-10419 and I understood that HDFS-on-HDSL
> integration does not necessarily require a lock split. However, there still
> needs to be refactoring to clearly define the FSN and BM interfaces and
> make the BM pluggable so HDSL can be swapped in. This is a major
> undertaking and risky. We did a similar refactoring in 2.x which made
> backports hard and introduced bugs. I don't think we should have done this
> in a minor release.
> 
> Furthermore, I don't know what your expectation is on how long it will take
> to stabilize HDSL, but this horizon for other storage systems is typically
> measured in years rather than months.
> 
> Both of these feel like Hadoop 4 items: a ways out yet.
> 
> Moving on, there is a non-trivial maintenance cost to having this new code
> in the code base. Ozone bugs become our bugs. Ozone dependencies become our
> dependencies. Ozone's security flaws are our security flaws. All of this
> negatively affects our already lumbering release schedule, and thus our
> ability to deliver and iterate on the features we're already trying to
> ship. Even if Ozone is separate and off by default, this is still a large
> amount of code that comes with a large maintenance cost. I don't want to
> incur this cost when the benefit is still a ways out.
> 
> We disagree on the necessity of sharing a repo and sharing operational
> behaviors. Libraries exist as a method for sharing code. HDFS also hardly
> has a monopoly on intermediating storage today. Disks are shared with MR
> shuffle, Spark/Impala spill, log output, Kudu, Kafka, etc. Operationally
> we've made this work. Having Ozone/HDSL in a separate process can even be
> seen as an operational advantage since it's isolated. I firmly believe that
> we can solve any implementation issues even with separate processes.
> 
> This is why I asked about making this a separate project. Given that these
> two efforts (HDSL stabilization and NN refactoring) are a ways out, the
> best way to get Ozone/HDSL in the hands of users today is to release it as
> its own project. Owen mentioned making a Hadoop subproject; we'd have to
> hash out what exactly this means (I assume a separate repo still managed by
> the Hadoop project), but I think we could make this work if it's more
> attractive than incubation or a new TLP.
> 
> I'm excited about the possibilities of both HDSL and the NN refactoring in
> ensuring a future for HDFS for years to come. A pluggable block manager
>