[jira] [Created] (HDFS-12285) Better handling of namenode ip address change

2017-08-09 Thread Ming Ma (JIRA) 
Ming Ma created HDFS-12285:
--

 Summary: Better handling of namenode ip address change
 Key: HDFS-12285
 URL: https://issues.apache.org/jira/browse/HDFS-12285
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ming Ma


RPC client layer provides functionality to detect ip address change:

{noformat}
Client.java
private synchronized boolean updateAddress() throws IOException {
  // Do a fresh lookup with the old host name.
  InetSocketAddress currentAddr = NetUtils.createSocketAddrForHost(
   server.getHostName(), server.getPort());
..
}
{noformat}

To use this feature, we need to enable retry via 
{{dfs.client.retry.policy.enabled}}. Otherwise {{TryOnceThenFail}} RetryPolicy 
will be used; which caused {{handleConnectionFailure}} to throw 
{{ConnectException}} exception without retrying with the new ip address.

{noformat}
private void handleConnectionFailure(int curRetries, IOException ioe
) throws IOException {
  closeConnection();

  final RetryAction action;
  try {
action = connectionRetryPolicy.shouldRetry(ioe, curRetries, 0, true);
  } catch(Exception e) {
throw e instanceof IOException? (IOException)e: new IOException(e);
  }
  ..
  }
{noformat}


However, using such configuration isn't ideal. What happens is DFSClient still 
has the cached the old ip address created by {{namenode = 
proxyInfo.getProxy();}}. Then when a new rpc connection is created, it starts 
with the old ip followed by retry with the new ip. It will be nice if DFSClient 
can refresh namenode proxy automatically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12284) Router support for Kerberos and delegation tokens

2017-08-09 Thread Zhe Zhang (JIRA) 
Zhe Zhang created HDFS-12284:


 Summary: Router support for Kerberos and delegation tokens
 Key: HDFS-12284
 URL: https://issues.apache.org/jira/browse/HDFS-12284
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: security
Reporter: Zhe Zhang
Assignee: xiangguang zheng


HDFS Router should support Kerberos authentication and issuing / managing HDFS 
delegation tokens.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog

2017-08-09 Thread Weiwei Yang (JIRA) 
Weiwei Yang created HDFS-12283:
--

 Summary: Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
 Key: HDFS-12283
 URL: https://issues.apache.org/jira/browse/HDFS-12283
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone, scm
Reporter: Weiwei Yang
Assignee: Weiwei Yang


The DeletedBlockLog is a persisted log in SCM to keep tracking container blocks 
which are under deletion. It maintains info about under-deletion container 
blocks that notified by KSM, and the state how it is processed. We can use 
RocksDB to implement the 1st version of the log, the schema looks like

||TxID||ContainerName||Block List||ProcessedCount||
|0|c1|b1,b2,b3|0|
|1|c2|b1|3|
|2|c2|b2, b3|-1|

Some explanations

# TxID is an incremental long value transaction ID for ONE container and 
multiple blocks
# Container name is the name of the container
# Block list is a list of block IDs
# ProcessedCount is the number of times SCM has sent this record to datanode, 
it represents the "state" of the transaction, it is in range of \[-1, 5\], -1 
means the transaction eventually failed after some retries, 5 is the max number 
times of retries.

We need to define {{DeletedBlockLog}} as an interface and implement this with 
RocksDB {{MetadataStore}} as the first version.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

RE: How to restore data from HDFS rm -skipTrash

2017-08-09 Thread Brahma Reddy Battula 
You can refer the following link also.

https://community.hortonworks.com/articles/26181/how-to-recover-accidentally-deleted-file-in-hdfs.html


--Brahma Reddy Battula

-Original Message-
From: Konstantin Shvachko [mailto:shv.had...@gmail.com] 
Sent: 05 August 2017 02:30
To: panfei
Cc: hdfs-dev@hadoop.apache.org
Subject: Re: How to restore data from HDFS rm -skipTrash

If you stopped the cluster immediately you may have a chance to restore most of 
the data.
It's a manual and hacky process.
You can use the Offline Edits Viewer to see the latest edits and cut off the 
delete transactions that was issues by mistake.
Then you can restart the namenode with the modifies edits log.
Hope this helps.

Thanks,
--Konstantin

On Thu, Aug 3, 2017 at 8:38 PM, panfei  wrote:

> -- Forwarded message --
> From: panfei 
> Date: 2017-08-04 11:23 GMT+08:00
> Subject: How to restore data from HDFS rm -skipTrash
> To: CDH Users 
>
>
> some one mistakenly do a rm -skipTrash operation on the HDFS, but we 
> stop the namenode and datanodes immediately. (CDH 5.4.5)
>
> I want to know is there any way to stop the deletion process ?
>
> and how ?
>
> thanks very in advance.
>

[jira] [Created] (HDFS-12282) Ozone: DeleteKey-4: SCM periodically sends block deletion message to datanode via HB and handles response

2017-08-09 Thread Weiwei Yang (JIRA) 
Weiwei Yang created HDFS-12282:
--

 Summary: Ozone: DeleteKey-4: SCM periodically sends block deletion 
message to datanode via HB and handles response
 Key: HDFS-12282
 URL: https://issues.apache.org/jira/browse/HDFS-12282
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, ozone, scm
Reporter: Weiwei Yang
Assignee: Weiwei Yang


This is the task 3 in the design doc, implements the SCM to datanode 
interactions. Including

# SCM sends block deletion message via HB to datanode
# datanode changes block state to deleting when processes the HB response
# datanode sends deletion ACKs back to SCM
# SCM handles ACKs and removes blocks in DB



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org