date:20150217

[jira] [Updated] (HDFS-7780) Update use of Iterator to Iterable in DataXceiverServer and SnapshotDiffInfo

2015-02-17 Thread Akira AJISAKA (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated HDFS-7780:

Summary: Update use of Iterator to Iterable in DataXceiverServer and 
SnapshotDiffInfo  (was: Update use of Iterator to Iterable)

 Update use of Iterator to Iterable in DataXceiverServer and SnapshotDiffInfo
 

 Key: HDFS-7780
 URL: https://issues.apache.org/jira/browse/HDFS-7780
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
 Attachments: HDFS-7780.001.patch, HDFS-7780.002.patch, 
 HDFS-7780.003.patch


 Found these using the IntelliJ Findbugs-IDEA plugin, which uses findbugs3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7188) support build libhdfs3 on windows

2015-02-17 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325149#comment-14325149
 ] 

Colin Patrick McCabe commented on HDFS-7188:


{code}
if (syscalls::getpeername(sock, peer, 
reinterpret_castint*(len))) {
{code}

This seems a bit concerning.  How do we know that {{int}} is the same length as 
{{socklen_t}}?  Why don't we just change the variable to be of type 
{{socklen_t}}?

{code}
49  #ifdef _WIN32
50  memcpy(clientId[0], id, sizeof(uuid_t));
51  #else
52  memcpy(clientId[0], id, sizeof(uuid_t));
53  #endif
{code}

I didn't look into this closely.  Why is this necessary?  Are we even using 
libuuid on Windows?  I almost think we should just drop the libuuid dependency 
either way, since it's basically just putting a random number into a 128-bit 
number... not exactly a very difficult thing to code (aside from the usual 
issues with getting good random numbers in C)

{{GetInitNamenodeIndex}}: I realize your patch didn't add this function.  But I 
still haven't figured out what the heck this is doing in the client.  The Java 
client doesn't do anything with files under /tmp to determine which NN to 
contact first... it just gets it from the configuration.  I'd prefer to just do 
what the Java client is doing here rather than implement this for N different 
OSes (although maybe we should have a follow-on JIRA).  Same with 
StackPrinter.cc...

 support build libhdfs3 on windows
 -

 Key: HDFS-7188
 URL: https://issues.apache.org/jira/browse/HDFS-7188
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
 Environment: Windows System, Visual Studio 2010
Reporter: Zhanwei Wang
Assignee: Thanh Do
 Attachments: HDFS-7188-branch-HDFS-6994-0.patch, 
 HDFS-7188-branch-HDFS-6994-1.patch, HDFS-7188-branch-HDFS-6994-2.patch


 libhdfs3 should work on windows



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7807) libhdfs htable.c: fix htable resizing, add unit test

2015-02-17 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325328#comment-14325328
 ] 

Hadoop QA commented on HDFS-7807:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12699353/HDFS-7807.001.patch
  against trunk revision 6dc8812.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.server.balancer.TestBalancer

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/9605//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9605//console

This message is automatically generated.

 libhdfs htable.c: fix htable resizing, add unit test
 

 Key: HDFS-7807
 URL: https://issues.apache.org/jira/browse/HDFS-7807
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: native
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
 Attachments: HDFS-7807.001.patch


 libhdfs htable.c: fix htable resizing, add unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client

2015-02-17 Thread Colin Patrick McCabe (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325127#comment-14325127
]

Colin Patrick McCabe commented on HDFS-6994:

[~wangzw], can you add some minimum compiler version guidelines to the README
in {{./hadoop-hdfs-project/hadoop-hdfs/src/contrib/libhdfs3/README.apt.vm}}?
This will help clear up some confusion, I bet.

I also think a lot of the exception stuff could be simplified a lot... we
should be following the Google style guide, as mentioned earlier, so exceptions
should not be used here. Probably having a catch (std::exception) with logging
and a catch (...) block is enough for the C APIs. That would also reduce the
number of esoteric compiler features we were using.

libhdfs3 - A native C/C++ HDFS client
-

Key: HDFS-6994
URL: https://issues.apache.org/jira/browse/HDFS-6994
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs-client
Reporter: Zhanwei Wang
Assignee: Zhanwei Wang
Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch

Hi All
I just got the permission to open source libhdfs3, which is a native C/C++
HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol.
libhdfs3 provide the libhdfs style C interface and a C++ interface. Support
both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos
authentication.
libhdfs3 is currently used by HAWQ of Pivotal
I'd like to integrate libhdfs3 into HDFS source code to benefit others.
You can find libhdfs3 code from github
https://github.com/PivotalRD/libhdfs3
http://pivotalrd.github.io/libhdfs3/

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2015-02-17 Thread Colin Patrick McCabe (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325162#comment-14325162
]

Colin Patrick McCabe commented on HDFS-7784:

At the end of the day, there are situations where you have to restart both
NameNodes. For example, you might have hit a bug that causes both the standby
and the active to crash. We've had bugs like that in the past. So I do think
this is an important improvement.

I think the discussion here has been a little too dismissive. Some people are
regularly spending 10 minutes to load their big fsimages... I don't think those
people would write off a 2x (or 2.5x speedup) as not good enough.

I do think [~wheat9]'s point about avoiding complexity is good. Can we get
some benefit just doing a really large amount of readahead? For example, if
we had a background thread that ran concurrently, that simply did nothing but
read the FSImage from start to back, it would warm up the buffer cache for
the other thread. This would mean that our single-threaded loading process
would spend less time waiting for disk I/O. Maybe try that out and see what
the numbers look like on a really big fsimage (something like 5-7 GB).

load fsimage in parallel

Key: HDFS-7784
URL: https://issues.apache.org/jira/browse/HDFS-7784
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Walter Su
Assignee: Walter Su
Priority: Minor
Attachments: HDFS-7784.001.patch, test-20150213.pdf

When single Namenode has huge amount of files, without using federation, the
startup/restart speed is slow. The fsimage loading step takes the most of the
time. fsimage loading can seperate to two parts, deserialization and object
construction(mostly map insertion). Deserialization takes the most of CPU
time. So we can do deserialization in parallel, and add to hashmap in serial.
It will significantly reduce the NN start time.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7648) Verify the datanode directory layout

2015-02-17 Thread Colin Patrick McCabe (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325196#comment-14325196
]

Colin Patrick McCabe commented on HDFS-7648:

bq. I see Colin that point that before we understand the problem, our system
should not be too smart fixing it. However, after we know the cause of the
problem (say, the admin moved some blocks manually), we need some way to fix
those misplaced blocks. How about adding a conf to enable/disable the auto-fix
feature and the default is disabled?

I wouldn't object to a configuration like that, but I also question whether it
is needed. Has this ever actually happened? And if it did happen, isn't the
answer more likely to be stop editing the VERSION file manually, silly or
your ext4 filesystem is bad and needs to be completely reformatted rather
than DN should cleverly fix?

bq. Colin Patrick McCabe kindly review the patch. Thanks!

We should be logging in the {{compileReport}} function, not a new function. We
can check whether the location is correct around the same place we're checking
the file name, etc.

Verify the datanode directory layout

Key: HDFS-7648
URL: https://issues.apache.org/jira/browse/HDFS-7648
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Tsz Wo Nicholas Sze
Assignee: Rakesh R
Attachments: HDFS-7648-3.patch, HDFS-7648-4.patch, HDFS-7648.patch,
HDFS-7648.patch

HDFS-6482 changed datanode layout to use block ID to determine the directory
to store the block. We should have some mechanism to verify it. Either
DirectoryScanner or block report generation could do the check.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7773) Additional metrics in HDFS to be accessed via jmx.

2015-02-17 Thread Akira AJISAKA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325215#comment-14325215
 ] 

Akira AJISAKA commented on HDFS-7773:
-

Hi [~anu], thank you for the patch. Would you document the new metrics to 
{{Metrics.md}}?

 Additional metrics in HDFS to be accessed via jmx.
 --

 Key: HDFS-7773
 URL: https://issues.apache.org/jira/browse/HDFS-7773
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: datanode, namenode
Reporter: Anu Engineer
Assignee: Anu Engineer
 Attachments: hdfs-7773.001.patch


 We would like to have the following metrics added to DataNode and name node 
 this to improve Ambari dashboard
 1) DN disk i/o utilization
 2) DN network i/o utilization
 3) Namenode read operations 
 4) Namenode write operations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7535) Utilize Snapshot diff report for distcp

2015-02-17 Thread Jing Zhao (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-7535:

Status: Patch Available  (was: Open)

 Utilize Snapshot diff report for distcp
 ---

 Key: HDFS-7535
 URL: https://issues.apache.org/jira/browse/HDFS-7535
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: distcp, snapshots
Reporter: Jing Zhao
Assignee: Jing Zhao
 Attachments: HDFS-7535.000.patch, HDFS-7535.001.patch


 Currently HDFS snapshot diff report can identify file/directory creation, 
 deletion, rename and modification under a snapshottable directory. We can use 
 the diff report for distcp between the primary cluster and a backup cluster 
 to avoid unnecessary data copy. This is especially useful when there is a big 
 directory rename happening in the primary cluster: the current distcp cannot 
 detect the rename op thus this rename usually leads to large amounts of real 
 data copy.
 More details of the approach will come in the first comment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7535) Utilize Snapshot diff report for distcp

2015-02-17 Thread Jing Zhao (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jing Zhao updated HDFS-7535:

Attachment: HDFS-7535.001.patch

Update the patch with new strategies to handle rename operations and also add
unit tests.

Currently for this feature we have the following assumptions:
# Both the source and target FileSystem must be DistributedFileSystem
# Two snapshots (e.g., s1 and s2) have been created on the source FS. The diff
between these two snapshots will be copied to the target FS.
# The target has the same snapshot s1. No changes have been made on the target
since s1. All the files/directories in the target are the same with source.s1

We verify these assumptions before the sync and we fallback to the default
distcp behavior if the assumptions do not stand. Note that for #3 currently we
only check the diff before the current target and target.s1 is empty, instead
of directly comparing target to source.s1. This may be fine since any failure
while applying the snapshot diff on the target will cause the distcp to copy
all the data.

The main challenge here is to translate the rename diffs to doable rename ops.
For example, if we have the following rename ops happening in the source:
1) /test -- /foo-tmp
2) /foo -- /test
3) /bar -- /foo
4) /foo-tmp -- /bar

The snapshot diff report now looks like:
R /foo -- /test
R /test -- /bar
R /bar -- /foo

This diff report cannot be directly applied. The current patch thus create a
tmp folder and breaks each rename op into two steps: move the source to the tmp
folder and then move the data from tmp to target. Then we only need to sort all
the first-phase renames based on the source paths (to make sure the files and
subdirs are moved before their parents/ancestors), and sort all the
second-phase renames based on the target paths (to make sure the parent
directories are created first).

Utilize Snapshot diff report for distcp
---

Key: HDFS-7535
URL: https://issues.apache.org/jira/browse/HDFS-7535
Project: Hadoop HDFS
Issue Type: Improvement
Components: distcp, snapshots
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-7535.000.patch, HDFS-7535.001.patch

Currently HDFS snapshot diff report can identify file/directory creation,
deletion, rename and modification under a snapshottable directory. We can use
the diff report for distcp between the primary cluster and a backup cluster
to avoid unnecessary data copy. This is especially useful when there is a big
directory rename happening in the primary cluster: the current distcp cannot
detect the rename op thus this rename usually leads to large amounts of real
data copy.
More details of the approach will come in the first comment.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7780) Update use of Iterator to Iterable in DataXceiverServer and SnapshotDiffInfo

2015-02-17 Thread Akira AJISAKA (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated HDFS-7780:

   Resolution: Fixed
Fix Version/s: 2.7.0
   Status: Resolved  (was: Patch Available)

I've committed this to trunk and branch-2. Thanks [~rchiang] for the 
contribution!

 Update use of Iterator to Iterable in DataXceiverServer and SnapshotDiffInfo
 

 Key: HDFS-7780
 URL: https://issues.apache.org/jira/browse/HDFS-7780
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
 Fix For: 2.7.0

 Attachments: HDFS-7780.001.patch, HDFS-7780.002.patch, 
 HDFS-7780.003.patch


 Found these using the IntelliJ Findbugs-IDEA plugin, which uses findbugs3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HDFS-7285) Erasure Coding Support inside HDFS

2015-02-17 Thread Harsh J (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Harsh J updated HDFS-7285:
--
Comment: was deleted

(was: I am will out of office for CN New Year from 2.15-2.26 , I may reply
e-mail slowly, please call me 13764370648 when there are urgent mater.
)

Erasure Coding Support inside HDFS
--

Key: HDFS-7285
URL: https://issues.apache.org/jira/browse/HDFS-7285
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Weihua Jiang
Assignee: Zhe Zhang
Attachments: ECAnalyzer.py, ECParser.py,
HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf,
HDFSErasureCodingDesign-20150204.pdf, HDFSErasureCodingDesign-20150206.pdf,
fsimage-analysis-20150105.pdf

Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice
of data reliability, comparing to the existing HDFS 3-replica approach. For
example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks,
with storage overhead only being 40%. This makes EC a quite attractive
alternative for big data storage, particularly for cold data.
Facebook had a related open source project called HDFS-RAID. It used to be
one of the contribute packages in HDFS but had been removed since Hadoop 2.0
for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends
on MapReduce to do encoding and decoding tasks; 2) it can only be used for
cold files that are intended not to be appended anymore; 3) the pure Java EC
coding implementation is extremely slow in practical use. Due to these, it
might not be a good idea to just bring HDFS-RAID back.
We (Intel and Cloudera) are working on a design to build EC into HDFS that
gets rid of any external dependencies, makes it self-contained and
independently maintained. This design lays the EC feature on the storage type
support and considers compatible with existing HDFS features like caching,
snapshot, encryption, high availability and etc. This design will also
support different EC coding schemes, implementations and policies for
different deployment scenarios. By utilizing advanced libraries (e.g. Intel
ISA-L library), an implementation can greatly improve the performance of EC
encoding/decoding and makes the EC solution even more attractive. We will
post the design document soon.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7807) libhdfs htable.c: fix htable resizing, add unit test

2015-02-17 Thread Colin Patrick McCabe (JIRA)

Colin Patrick McCabe created HDFS-7807:
--

 Summary: libhdfs htable.c: fix htable resizing, add unit test
 Key: HDFS-7807
 URL: https://issues.apache.org/jira/browse/HDFS-7807
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: native
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe


libhdfs htable.c: fix htable resizing, add unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7019) Add unit test for libhdfs3

2015-02-17 Thread Thanh Do (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325098#comment-14325098
 ] 

Thanh Do commented on HDFS-7019:


Hi [~wangzw],

Is there a specific reason that we can not use existing unit tests that you 
already wrote?

 Add unit test for libhdfs3
 --

 Key: HDFS-7019
 URL: https://issues.apache.org/jira/browse/HDFS-7019
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Zhanwei Wang
 Attachments: HDFS-7019.patch


 Add unit test for libhdfs3



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7397) The conf key dfs.client.read.shortcircuit.streams.cache.size is misleading

2015-02-17 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325216#comment-14325216
 ] 

Colin Patrick McCabe commented on HDFS-7397:


Why not just add some text to the description in {{hdfs-default.xml}}?

 The conf key dfs.client.read.shortcircuit.streams.cache.size is misleading
 

 Key: HDFS-7397
 URL: https://issues.apache.org/jira/browse/HDFS-7397
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Reporter: Tsz Wo Nicholas Sze
Assignee: Brahma Reddy Battula
Priority: Minor

 For dfs.client.read.shortcircuit.streams.cache.size, is it in MB or KB?  
 Interestingly, it is neither in MB nor KB.  It is the number of shortcircuit 
 streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7535) Utilize Snapshot diff report for distcp

2015-02-17 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325295#comment-14325295
 ] 

Hadoop QA commented on HDFS-7535:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12699392/HDFS-7535.001.patch
  against trunk revision 685af8a.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 1156 javac 
compiler warnings (more than the trunk's current 1155 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-distcp.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/9606//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HDFS-Build/9606//artifact/patchprocess/newPatchFindbugsWarningshadoop-distcp.html
Javac warnings: 
https://builds.apache.org/job/PreCommit-HDFS-Build/9606//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9606//console

This message is automatically generated.

 Utilize Snapshot diff report for distcp
 ---

 Key: HDFS-7535
 URL: https://issues.apache.org/jira/browse/HDFS-7535
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: distcp, snapshots
Reporter: Jing Zhao
Assignee: Jing Zhao
 Attachments: HDFS-7535.000.patch, HDFS-7535.001.patch


 Currently HDFS snapshot diff report can identify file/directory creation, 
 deletion, rename and modification under a snapshottable directory. We can use 
 the diff report for distcp between the primary cluster and a backup cluster 
 to avoid unnecessary data copy. This is especially useful when there is a big 
 directory rename happening in the primary cluster: the current distcp cannot 
 detect the rename op thus this rename usually leads to large amounts of real 
 data copy.
 More details of the approach will come in the first comment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7807) libhdfs htable.c: fix htable resizing, add unit test

2015-02-17 Thread Colin Patrick McCabe (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HDFS-7807:
---
Status: Patch Available  (was: Open)

 libhdfs htable.c: fix htable resizing, add unit test
 

 Key: HDFS-7807
 URL: https://issues.apache.org/jira/browse/HDFS-7807
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: native
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
 Attachments: HDFS-7807.001.patch


 libhdfs htable.c: fix htable resizing, add unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7759) Provide existence-of-a-second-file implementation for pinning blocks on Datanode

2015-02-17 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325105#comment-14325105
 ] 

Colin Patrick McCabe commented on HDFS-7759:


This seems like it will lead to a lot of problems.  If blocks can't be moved, 
then the balancer can't work.  It means that if nodes are lost in the cluster, 
then we can't re-replicate.  Why not simply write a pluggable block placement 
policy instead?

 Provide existence-of-a-second-file implementation for pinning blocks on 
 Datanode
 

 Key: HDFS-7759
 URL: https://issues.apache.org/jira/browse/HDFS-7759
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: datanode
Reporter: zhaoyunjiong
Assignee: zhaoyunjiong
 Attachments: HDFS-7759.patch


 Provide existence-of-a-second-file implementation for pinning blocks on 
 Datanode  and let admin choosing the mechanism(use sticky bit or 
 existence-of-a-second-file) to pinning blocks on favored Datanode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-4266) BKJM: Separate write and ack quorum

2015-02-17 Thread Rakesh R (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325375#comment-14325375
 ] 

Rakesh R commented on HDFS-4266:


Thanks a lot [~umamaheswararao] for the reviews and committing the patch.

 BKJM: Separate write and ack quorum
 ---

 Key: HDFS-4266
 URL: https://issues.apache.org/jira/browse/HDFS-4266
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha
Reporter: Ivan Kelly
Assignee: Rakesh R
 Fix For: 2.7.0

 Attachments: 001-HDFS-4266.patch, 002-HDFS-4266.patch, 
 003-HDFS-4266.patch, 004-HDFS-4266.patch, 005-HDFS-4266.patch


 BOOKKEEPER-208 allows the ack and write quorums to be different sizes to 
 allow writes to be unaffected by any bookie failure. BKJM should be able to 
 take advantage of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 >

1 - 100 of 144 matches

Mail list logo