[jira] [Updated] (HDFS-15155) writeIoRate of DataNodeVolumeMetrics is never used
[
https://issues.apache.org/jira/browse/HDFS-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibin Huang updated HDFS-15155:
Attachment: HDFS-15155.004.patch
> writeIoRate of DataNodeVolumeMetrics is never used
> --
>
> Key: HDFS-15155
> URL: https://issues.apache.org/jira/browse/HDFS-15155
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HDFS-15155.001.patch, HDFS-15155.002.patch,
> HDFS-15155.003.patch, HDFS-15155.004.patch
>
>
> There is some incorrect object using in DataNodeVolumeMetrics, writeIoRate is
> never used and syncIoRate should be replaced by writeIoRate in the following
> code:
> {code:java}
> // Based on writeIoRate
> public long getWriteIoSampleCount() {
> return syncIoRate.lastStat().numSamples();
> }
> public double getWriteIoMean() {
> return syncIoRate.lastStat().mean();
> }
> public double getWriteIoStdDev() {
> return syncIoRate.lastStat().stddev();
> }
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10499) TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently
[
https://issues.apache.org/jira/browse/HDFS-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051345#comment-17051345
]
Ahmed Hussein commented on HDFS-10499:
--
[~linyiqun] , [~brahmareddy]. I ported the fix to branch-2.10. Can you please
take a look at the patch [^HDFS-10499-branch-2.10.001.patch] ?
> TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails
> Intermittently
> -
>
> Key: HDFS-10499
> URL: https://issues.apache.org/jira/browse/HDFS-10499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, test
>Affects Versions: 3.0.0-alpha1
>Reporter: Hanisha Koneru
>Assignee: Yiqun Lin
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10499-branch-2.10.001.patch, HDFS-10499.001.patch,
> HDFS-10499.002.patch
>
>
> Per https://builds.apache.org/job/PreCommit-HDFS-Build/15646/testReport/, we
> had the following failure. Local rerun is successful.
> Stack Trace:
> {panel}
> java.lang.AssertionError: expected:<17> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency.testGenerationStampInFuture(TestNameNodeMetadataConsistency.java:113)
> {panel}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15203) A bug in ViewFileSystemBaseTest
[
https://issues.apache.org/jira/browse/HDFS-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051342#comment-17051342
]
Ravuri Sushma sree commented on HDFS-15203:
---
Hi [~kihwal]
In believe the intention behind this Unit Test was getting block locations
using the following two overloaded methods in FileSystem.java
{code:java}
public BlockLocation[] getFileBlockLocations(FileStatus file,
long start, long len) throws IOException {{code}
and
{code:java}
public BlockLocation[] getFileBlockLocations(Path p,
long start, long len) throws IOException {{code}
Coming to ViewFsBaseTest, there is no such getFileBlockLocations method in
FileContext.java accepting FileStatus as parameter,
{code:java}
// Same test but now get it via the FileStatus Parameter
fcView.getFileBlockLocations(viewFilePath, 0, 10240+100);
targetBL = fcTarget.getFileBlockLocations(targetFilePath, 0, 10240+100);{code}
Looks redundant and in my opinion it should be removed.
> A bug in ViewFileSystemBaseTest
> ---
>
> Key: HDFS-15203
> URL: https://issues.apache.org/jira/browse/HDFS-15203
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Priority: Trivial
>
> Missing an assignment here:
> {code:java}
> @Test
> public void testGetBlockLocations() throws IOException {
> ...
> // Same test but now get it via the FileStatus Parameter
> fsView.getFileBlockLocations(
> fsView.getFileStatus(viewFilePath), 0, 10240+100);
> targetBL = fsTarget.getFileBlockLocations(
> fsTarget.getFileStatus(targetFilePath), 0, 10240+100);
> compareBLs(viewBL, targetBL);
> {code}
> But more importantly, I am not sure what is the difference between this and
> the previous check. Are they redundant?
> {code:java}
> BlockLocation[] viewBL =
> fsView.getFileBlockLocations(fsView.getFileStatus(viewFilePath), 0,
> 10240+100);
> Assert.assertEquals(SupportsBlocks ? 10 : 1, viewBL.length);
> BlockLocation[] targetBL =
> fsTarget.getFileBlockLocations(fsTarget.getFileStatus(targetFilePath), 0,
> 10240+100);
> compareBLs(viewBL, targetBL);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15196) RBF: RouterRpcServer getListing cannot list large dirs correctly
[
https://issues.apache.org/jira/browse/HDFS-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051204#comment-17051204
]
Jinglun commented on HDFS-15196:
Hi [~fengnanli], thanks your nice work !
{quote}I guess there is a reason that the mount point is appended as a dir
{quote}
In the linux file system concept the mount point is a directory. So keep the
same as linux would be a good choice. Besides I don't see extra benefits if we
enable the router mount point as a file.
Other suggestions would be may be we can have 2 test cases, one for the sorted
order and one for the infinite loop.
> RBF: RouterRpcServer getListing cannot list large dirs correctly
>
>
> Key: HDFS-15196
> URL: https://issues.apache.org/jira/browse/HDFS-15196
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Critical
> Attachments: HDFS-15196.001.patch, HDFS-15196.002.patch,
> HDFS-15196.003.patch, HDFS-15196.003.patch, HDFS-15196.004.patch,
> HDFS-15196.005.patch
>
>
> In RouterRpcServer, getListing function is handled as two parts:
> # Union all partial listings from destination ns + paths
> # Append mount points for the dir to be listed
> In the case of large dir which is bigger than DFSConfigKeys.DFS_LIST_LIMIT
> (with default value 1k), the batch listing will be used and the startAfter
> will be used to define the boundary of each batch listing. However, step 2
> here will add existing mount points, which will mess up with the boundary of
> the batch, thus making the next batch startAfter wrong.
> The fix is just to append the mount points when there is no more batch query
> necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15206) HDFS synchronous reads from local file system
[
https://issues.apache.org/jira/browse/HDFS-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mania Abdi updated HDFS-15206:
--
Priority: Minor (was: Major)
> HDFS synchronous reads from local file system
> -
>
> Key: HDFS-15206
> URL: https://issues.apache.org/jira/browse/HDFS-15206
> Project: Hadoop HDFS
> Issue Type: Improvement
> Environment: !Screenshot from 2020-03-03 22-07-26.png!
>Reporter: Mania Abdi
>Priority: Minor
> Attachments: Screenshot from 2020-03-03 22-07-26.png
>
>
> Hello everyone,
> I ran a simple benchmark with runs ``` hadoop fs -get /file1.txt ```, and
> file1.txt has 1MB size and I capture the workflow of requests using XTrace.
> By evaluating the workflow trace, I noticed that datanode issues 64KB
> synchronous read request to local file system to read the data, and sends the
> data back and waits for completion. I had a code walk over HDFS code to
> verify the pattern and it was correct. I want to have two suggestions, (1)
> since each file in HDFS block size is usually 128MB, We could use the mmap
> mapping via FileChannel class to load the file into memory and enable file
> system prefetching and look ahead in background, instead of synchronously
> reading from disk. The second suggestion is to use asynchronous read
> operations to local disk of the datanode. I was wondering if there is a logic
> behind synchronous reads from the file system?
>
> Code:
> $HADOOP_SRC/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
> line 586
> {code:java}
> /**
>* Sends a packet with up to maxChunks chunks of data.
>*
>* @param pkt buffer used for writing packet data
>* @param maxChunks maximum number of chunks to send
>* @param out stream to send data to
>* @param transferTo use transferTo to send data
>* @param throttler used for throttling data transfer bandwidth
>*/
> private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
> boolean transferTo, DataTransferThrottler throttler) throws IOException
> {
> int dataLen = (int) Math.min(endOffset - offset,
> (chunkSize * (long) maxChunks));
>
> int numChunks = numberOfChunks(dataLen); // Number of chunks be sent in
> the packet
> int checksumDataLen = numChunks * checksumSize;
> int packetLen = dataLen + checksumDataLen + 4;
> boolean lastDataPacket = offset + dataLen == endOffset && dataLen > 0;
> // The packet buffer is organized as follows:
> // ___D?D?D?D?
> //^ ^
> //| \ checksumOff
> //\ headerOff
> // _ padding, since the header is variable-length
> // H = header and length prefixes
> // C = checksums
> // D? = data, if transferTo is false.
>
> int headerLen = writePacketHeader(pkt, dataLen, packetLen);
>
> // Per above, the header doesn't start at the beginning of the
> // buffer
> int headerOff = pkt.position() - headerLen;
>
> int checksumOff = pkt.position();
> byte[] buf = pkt.array();
>
> if (checksumSize > 0 && checksumIn != null) {
> readChecksum(buf, checksumOff, checksumDataLen); // write in
> progress that we need to use to get last checksum
> if (lastDataPacket && lastChunkChecksum != null) {
> int start = checksumOff + checksumDataLen - checksumSize;
> byte[] updatedChecksum = lastChunkChecksum.getChecksum();
>
> if (updatedChecksum != null) {
> System.arraycopy(updatedChecksum, 0, buf, start, checksumSize);
> }
> }
> }
>
> int dataOff = checksumOff + checksumDataLen;
> if (!transferTo) { // normal transfer
> IOUtils.readFully(blockIn, buf, dataOff, dataLen);
> if (verifyChecksum) {
> verifyChecksum(buf, dataOff, dataLen, numChunks, checksumOff);
> }
> }
>
> try {
> if (transferTo) {
> SocketOutputStream sockOut = (SocketOutputStream)out;
> // First write header and checksums
> sockOut.write(buf, headerOff, dataOff - headerOff);
>
> // no need to flush since we know out is not a buffered stream
> FileChannel fileCh = ((FileInputStream)blockIn).getChannel();
> LongWritable waitTime = new LongWritable();
> LongWritable transferTime = new LongWritable();
> sockOut.transferToFully(fileCh, blockInPosition, dataLen,
> waitTime, transferTime);
>
> datanode.metrics.addSendDataPacketBlockedOnNetworkNanos(waitTime.get());
> datanode.metrics.addSendDataPacketTransferNanos(transferTime.get());
> blockInPosition += dataLen;
> } else {
> // normal transfer
>
[jira] [Updated] (HDFS-15206) HDFS synchronous reads from local file system
[
https://issues.apache.org/jira/browse/HDFS-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mania Abdi updated HDFS-15206:
--
Component/s: datanode
> HDFS synchronous reads from local file system
> -
>
> Key: HDFS-15206
> URL: https://issues.apache.org/jira/browse/HDFS-15206
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Environment: !Screenshot from 2020-03-03 22-07-26.png!
>Reporter: Mania Abdi
>Priority: Minor
> Attachments: Screenshot from 2020-03-03 22-07-26.png
>
>
> Hello everyone,
> I ran a simple benchmark with runs ``` hadoop fs -get /file1.txt ```, and
> file1.txt has 1MB size and I capture the workflow of requests using XTrace.
> By evaluating the workflow trace, I noticed that datanode issues 64KB
> synchronous read request to local file system to read the data, and sends the
> data back and waits for completion. I had a code walk over HDFS code to
> verify the pattern and it was correct. I want to have two suggestions, (1)
> since each file in HDFS block size is usually 128MB, We could use the mmap
> mapping via FileChannel class to load the file into memory and enable file
> system prefetching and look ahead in background, instead of synchronously
> reading from disk. The second suggestion is to use asynchronous read
> operations to local disk of the datanode. I was wondering if there is a logic
> behind synchronous reads from the file system?
>
> Code:
> $HADOOP_SRC/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
> line 586
> {code:java}
> /**
>* Sends a packet with up to maxChunks chunks of data.
>*
>* @param pkt buffer used for writing packet data
>* @param maxChunks maximum number of chunks to send
>* @param out stream to send data to
>* @param transferTo use transferTo to send data
>* @param throttler used for throttling data transfer bandwidth
>*/
> private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
> boolean transferTo, DataTransferThrottler throttler) throws IOException
> {
> int dataLen = (int) Math.min(endOffset - offset,
> (chunkSize * (long) maxChunks));
>
> int numChunks = numberOfChunks(dataLen); // Number of chunks be sent in
> the packet
> int checksumDataLen = numChunks * checksumSize;
> int packetLen = dataLen + checksumDataLen + 4;
> boolean lastDataPacket = offset + dataLen == endOffset && dataLen > 0;
> // The packet buffer is organized as follows:
> // ___D?D?D?D?
> //^ ^
> //| \ checksumOff
> //\ headerOff
> // _ padding, since the header is variable-length
> // H = header and length prefixes
> // C = checksums
> // D? = data, if transferTo is false.
>
> int headerLen = writePacketHeader(pkt, dataLen, packetLen);
>
> // Per above, the header doesn't start at the beginning of the
> // buffer
> int headerOff = pkt.position() - headerLen;
>
> int checksumOff = pkt.position();
> byte[] buf = pkt.array();
>
> if (checksumSize > 0 && checksumIn != null) {
> readChecksum(buf, checksumOff, checksumDataLen); // write in
> progress that we need to use to get last checksum
> if (lastDataPacket && lastChunkChecksum != null) {
> int start = checksumOff + checksumDataLen - checksumSize;
> byte[] updatedChecksum = lastChunkChecksum.getChecksum();
>
> if (updatedChecksum != null) {
> System.arraycopy(updatedChecksum, 0, buf, start, checksumSize);
> }
> }
> }
>
> int dataOff = checksumOff + checksumDataLen;
> if (!transferTo) { // normal transfer
> IOUtils.readFully(blockIn, buf, dataOff, dataLen);
> if (verifyChecksum) {
> verifyChecksum(buf, dataOff, dataLen, numChunks, checksumOff);
> }
> }
>
> try {
> if (transferTo) {
> SocketOutputStream sockOut = (SocketOutputStream)out;
> // First write header and checksums
> sockOut.write(buf, headerOff, dataOff - headerOff);
>
> // no need to flush since we know out is not a buffered stream
> FileChannel fileCh = ((FileInputStream)blockIn).getChannel();
> LongWritable waitTime = new LongWritable();
> LongWritable transferTime = new LongWritable();
> sockOut.transferToFully(fileCh, blockInPosition, dataLen,
> waitTime, transferTime);
>
> datanode.metrics.addSendDataPacketBlockedOnNetworkNanos(waitTime.get());
> datanode.metrics.addSendDataPacketTransferNanos(transferTime.get());
> blockInPosition += dataLen;
> } else {
> // normal
[jira] [Updated] (HDFS-10499) TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently
[
https://issues.apache.org/jira/browse/HDFS-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated HDFS-10499:
-
Attachment: HDFS-10499-branch-2.10.001.patch
> TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails
> Intermittently
> -
>
> Key: HDFS-10499
> URL: https://issues.apache.org/jira/browse/HDFS-10499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, test
>Affects Versions: 3.0.0-alpha1
>Reporter: Hanisha Koneru
>Assignee: Yiqun Lin
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10499-branch-2.10.001.patch, HDFS-10499.001.patch,
> HDFS-10499.002.patch
>
>
> Per https://builds.apache.org/job/PreCommit-HDFS-Build/15646/testReport/, we
> had the following failure. Local rerun is successful.
> Stack Trace:
> {panel}
> java.lang.AssertionError: expected:<17> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency.testGenerationStampInFuture(TestNameNodeMetadataConsistency.java:113)
> {panel}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15206) HDFS synchronous reads from local file system
[
https://issues.apache.org/jira/browse/HDFS-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mania Abdi updated HDFS-15206:
--
Labels: performance (was: )
> HDFS synchronous reads from local file system
> -
>
> Key: HDFS-15206
> URL: https://issues.apache.org/jira/browse/HDFS-15206
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Environment: !Screenshot from 2020-03-03 22-07-26.png!
>Reporter: Mania Abdi
>Priority: Minor
> Labels: performance
> Attachments: Screenshot from 2020-03-03 22-07-26.png
>
>
> Hello everyone,
> I ran a simple benchmark with runs ``` hadoop fs -get /file1.txt ```, and
> file1.txt has 1MB size and I capture the workflow of requests using XTrace.
> By evaluating the workflow trace, I noticed that datanode issues 64KB
> synchronous read request to local file system to read the data, and sends the
> data back and waits for completion. I had a code walk over HDFS code to
> verify the pattern and it was correct. I want to have two suggestions, (1)
> since each file in HDFS block size is usually 128MB, We could use the mmap
> mapping via FileChannel class to load the file into memory and enable file
> system prefetching and look ahead in background, instead of synchronously
> reading from disk. The second suggestion is to use asynchronous read
> operations to local disk of the datanode. I was wondering if there is a logic
> behind synchronous reads from the file system?
>
> Code:
> $HADOOP_SRC/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
> line 586
> {code:java}
> /**
>* Sends a packet with up to maxChunks chunks of data.
>*
>* @param pkt buffer used for writing packet data
>* @param maxChunks maximum number of chunks to send
>* @param out stream to send data to
>* @param transferTo use transferTo to send data
>* @param throttler used for throttling data transfer bandwidth
>*/
> private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
> boolean transferTo, DataTransferThrottler throttler) throws IOException
> {
> int dataLen = (int) Math.min(endOffset - offset,
> (chunkSize * (long) maxChunks));
>
> int numChunks = numberOfChunks(dataLen); // Number of chunks be sent in
> the packet
> int checksumDataLen = numChunks * checksumSize;
> int packetLen = dataLen + checksumDataLen + 4;
> boolean lastDataPacket = offset + dataLen == endOffset && dataLen > 0;
> // The packet buffer is organized as follows:
> // ___D?D?D?D?
> //^ ^
> //| \ checksumOff
> //\ headerOff
> // _ padding, since the header is variable-length
> // H = header and length prefixes
> // C = checksums
> // D? = data, if transferTo is false.
>
> int headerLen = writePacketHeader(pkt, dataLen, packetLen);
>
> // Per above, the header doesn't start at the beginning of the
> // buffer
> int headerOff = pkt.position() - headerLen;
>
> int checksumOff = pkt.position();
> byte[] buf = pkt.array();
>
> if (checksumSize > 0 && checksumIn != null) {
> readChecksum(buf, checksumOff, checksumDataLen); // write in
> progress that we need to use to get last checksum
> if (lastDataPacket && lastChunkChecksum != null) {
> int start = checksumOff + checksumDataLen - checksumSize;
> byte[] updatedChecksum = lastChunkChecksum.getChecksum();
>
> if (updatedChecksum != null) {
> System.arraycopy(updatedChecksum, 0, buf, start, checksumSize);
> }
> }
> }
>
> int dataOff = checksumOff + checksumDataLen;
> if (!transferTo) { // normal transfer
> IOUtils.readFully(blockIn, buf, dataOff, dataLen);
> if (verifyChecksum) {
> verifyChecksum(buf, dataOff, dataLen, numChunks, checksumOff);
> }
> }
>
> try {
> if (transferTo) {
> SocketOutputStream sockOut = (SocketOutputStream)out;
> // First write header and checksums
> sockOut.write(buf, headerOff, dataOff - headerOff);
>
> // no need to flush since we know out is not a buffered stream
> FileChannel fileCh = ((FileInputStream)blockIn).getChannel();
> LongWritable waitTime = new LongWritable();
> LongWritable transferTime = new LongWritable();
> sockOut.transferToFully(fileCh, blockInPosition, dataLen,
> waitTime, transferTime);
>
> datanode.metrics.addSendDataPacketBlockedOnNetworkNanos(waitTime.get());
> datanode.metrics.addSendDataPacketTransferNanos(transferTime.get());
> blockInPosition +=
[jira] [Commented] (HDFS-15200) Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage
[
https://issues.apache.org/jira/browse/HDFS-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051279#comment-17051279
]
Hadoop QA commented on HDFS-15200:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
36s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 58s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 579 new + 0 unchanged -
0 fixed = 579 total (was 0) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 51s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestEncryptionZones |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.7 Server=19.03.7 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15200 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12995533/HDFS-15200-03.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux e9b95aea1d06 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bbd704b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/28894/artifact/out/branch-mvninstall-root.txt
|
| compile |
[jira] [Commented] (HDFS-14442) Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId
[
https://issues.apache.org/jira/browse/HDFS-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051284#comment-17051284
]
Ravuri Sushma sree commented on HDFS-14442:
---
[~surendrasingh] Can you please review
> Disagreement between HAUtil.getAddressOfActive and
> RpcInvocationHandler.getConnectionId
> ---
>
> Key: HDFS-14442
> URL: https://issues.apache.org/jira/browse/HDFS-14442
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Ravuri Sushma sree
>Priority: Major
> Attachments: HDFS-14442.001.patch, HDFS-14442.002.patch,
> HDFS-14442.003.patch, HDFS-14442.004.patch
>
>
> While working on HDFS-14245, we noticed a discrepancy in some proxy-handling
> code.
> The description of {{RpcInvocationHandler.getConnectionId()}} states:
> {code}
> /**
>* Returns the connection id associated with the InvocationHandler instance.
>* @return ConnectionId
>*/
> ConnectionId getConnectionId();
> {code}
> It does not make any claims about whether this connection ID will be an
> active proxy or not. Yet in {{HAUtil}} we have:
> {code}
> /**
>* Get the internet address of the currently-active NN. This should rarely
> be
>* used, since callers of this method who connect directly to the NN using
> the
>* resulting InetSocketAddress will not be able to connect to the active NN
> if
>* a failover were to occur after this method has been called.
>*
>* @param fs the file system to get the active address of.
>* @return the internet address of the currently-active NN.
>* @throws IOException if an error occurs while resolving the active NN.
>*/
> public static InetSocketAddress getAddressOfActive(FileSystem fs)
> throws IOException {
> if (!(fs instanceof DistributedFileSystem)) {
> throw new IllegalArgumentException("FileSystem " + fs + " is not a
> DFS.");
> }
> // force client address resolution.
> fs.exists(new Path("/"));
> DistributedFileSystem dfs = (DistributedFileSystem) fs;
> DFSClient dfsClient = dfs.getClient();
> return RPC.getServerAddress(dfsClient.getNamenode());
> }
> {code}
> Where the call {{RPC.getServerAddress()}} eventually terminates into
> {{RpcInvocationHandler#getConnectionId()}}, via {{RPC.getServerAddress()}} ->
> {{RPC.getConnectionIdForProxy()}} ->
> {{RpcInvocationHandler#getConnectionId()}}. {{HAUtil}} appears to be making
> an incorrect assumption that {{RpcInvocationHandler}} will necessarily return
> an _active_ connection ID. {{ObserverReadProxyProvider}} demonstrates a
> counter-example to this, since the current connection ID may be pointing at,
> for example, an Observer NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15204) TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051392#comment-17051392
]
Ahmed Hussein commented on HDFS-15204:
--
Thanks [~elgoiri] for the review.
I ran the JUnit inside a loop without any problem.
Can you please merge both patches whenever you have the chance to?
> TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
> ---
>
> Key: HDFS-15204
> URL: https://issues.apache.org/jira/browse/HDFS-15204
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: HDFS-15204-branch-2.10.001.patch, HDFS-15204.001.patch
>
>
> {code:bash}
> testRemoveCacheDescriptor:
> java.lang.AssertionError: CacheHit: 0, 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1344)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testRemoveCacheDescriptor(TestRetryCacheWithHA.java:1245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14977) Quota Usage and Content summary are not same in Truncate with Snapshot
[ https://issues.apache.org/jira/browse/HDFS-14977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14977: --- Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Quota Usage and Content summary are not same in Truncate with Snapshot > --- > > Key: HDFS-14977 > URL: https://issues.apache.org/jira/browse/HDFS-14977 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14977.001.patch, HDFS-14977.002.patch, > HDFS-14977.003.patch > > > steps : hdfs dfs -mkdir /dir > hdfs dfs -put file /dir (file size = 10bytes) > hdfs dfsadmin -allowSnapshot /dir > hdfs dfs -createSnapshot /dir s1 > space consumed with Quotausage and Content Summary is 30bytes > hdfs dfs -truncate -w 5 /dir/file > space consumed with Quotausage , Content Summary is 45 bytes > hdfs dfs -deleteSnapshot /dir s1 > space consumed with Quotausage is 45bytes and Content Summary is 15bytes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15155) writeIoRate of DataNodeVolumeMetrics is never used
[
https://issues.apache.org/jira/browse/HDFS-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051385#comment-17051385
]
Hadoop QA commented on HDFS-15155:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 59s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestEncryptionZones |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.7 Server=19.03.7 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15155 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12995609/HDFS-15155.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c187a7c0a28d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bbd704b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28895/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28895/testReport/ |
| Max. process+thread count | 3208 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28895/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (HDFS-15202) HDFS-client: boost ShortCircuit Cache
[
https://issues.apache.org/jira/browse/HDFS-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051452#comment-17051452
]
Danil Lipovoy commented on HDFS-15202:
--
[~weichiu]
ok, [done|https://github.com/apache/hadoop-common/pull/51])
> HDFS-client: boost ShortCircuit Cache
> -
>
> Key: HDFS-15202
> URL: https://issues.apache.org/jira/browse/HDFS-15202
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient
> Environment: 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 8 RegionServers (2 by host)
> 8 tables by 64 regions by 1.88 Gb data in each = 900 Gb total
> Random read in 800 threads via YCSB and a little bit updates (10% of reads)
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: hdfs_cpu.png, hdfs_reads.png
>
>
> I want to propose how to improve reading performance HDFS-client. The idea:
> create few instances ShortCircuit caches instead of one.
> The key points:
> 1. Create array of caches (set by
> clientShortCircuitNum=*dfs.client.short.circuit.num*, see in the pull
> requests below):
> {code:java}
> private ClientContext(String name, DfsClientConf conf, Configuration config) {
> ...
> shortCircuitCache = new ShortCircuitCache[this.clientShortCircuitNum];
> for (int i = 0; i < this.clientShortCircuitNum; i++) {
> this.shortCircuitCache[i] = ShortCircuitCache.fromConf(scConf);
> }
> {code}
> 2 Then divide blocks by caches:
> {code:java}
> public ShortCircuitCache getShortCircuitCache(long idx) {
> return shortCircuitCache[(int) (idx % clientShortCircuitNum)];
> }
> {code}
> 3. And how to call it:
> {code:java}
> ShortCircuitCache cache =
> clientContext.getShortCircuitCache(block.getBlockId());
> {code}
> The last number of offset evenly distributed from 0 to 9 - that's why all
> caches will full approximately the same.
> It is good for performance. Below the attachment, it is load test reading
> HDFS via HBase where clientShortCircuitNum = 1 vs 3. We can see that
> performance grows ~30%, CPU usage about +15%.
> Hope it is interesting for someone.
> Ready to explain some unobvious things.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15155) writeIoRate of DataNodeVolumeMetrics is never used
[
https://issues.apache.org/jira/browse/HDFS-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051487#comment-17051487
]
Íñigo Goiri commented on HDFS-15155:
The failed unit tests are unrelated.
+1 on [^HDFS-15155.004.patch].
> writeIoRate of DataNodeVolumeMetrics is never used
> --
>
> Key: HDFS-15155
> URL: https://issues.apache.org/jira/browse/HDFS-15155
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HDFS-15155.001.patch, HDFS-15155.002.patch,
> HDFS-15155.003.patch, HDFS-15155.004.patch
>
>
> There is some incorrect object using in DataNodeVolumeMetrics, writeIoRate is
> never used and syncIoRate should be replaced by writeIoRate in the following
> code:
> {code:java}
> // Based on writeIoRate
> public long getWriteIoSampleCount() {
> return syncIoRate.lastStat().numSamples();
> }
> public double getWriteIoMean() {
> return syncIoRate.lastStat().mean();
> }
> public double getWriteIoStdDev() {
> return syncIoRate.lastStat().stddev();
> }
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15204) TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051500#comment-17051500
]
Hudson commented on HDFS-15204:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18025 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18025/])
HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails (inigoiri: rev
1d4d0fcbe1e352b95c93dc016d3674db1126fb25)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
> TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
> ---
>
> Key: HDFS-15204
> URL: https://issues.apache.org/jira/browse/HDFS-15204
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 2.10.1
>
> Attachments: HDFS-15204-branch-2.10.001.patch, HDFS-15204.001.patch
>
>
> {code:bash}
> testRemoveCacheDescriptor:
> java.lang.AssertionError: CacheHit: 0, 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1344)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testRemoveCacheDescriptor(TestRetryCacheWithHA.java:1245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14977) Quota Usage and Content summary are not same in Truncate with Snapshot
[ https://issues.apache.org/jira/browse/HDFS-14977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051512#comment-17051512 ] Hudson commented on HDFS-14977: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18026 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18026/]) HDFS-14977. Quota Usage and Content summary are not same in Truncate (inigoiri: rev 3afd4cbe89eb79c12465077a2f55949a800f32ae) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java > Quota Usage and Content summary are not same in Truncate with Snapshot > --- > > Key: HDFS-14977 > URL: https://issues.apache.org/jira/browse/HDFS-14977 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14977.001.patch, HDFS-14977.002.patch, > HDFS-14977.003.patch > > > steps : hdfs dfs -mkdir /dir > hdfs dfs -put file /dir (file size = 10bytes) > hdfs dfsadmin -allowSnapshot /dir > hdfs dfs -createSnapshot /dir s1 > space consumed with Quotausage and Content Summary is 30bytes > hdfs dfs -truncate -w 5 /dir/file > space consumed with Quotausage , Content Summary is 45 bytes > hdfs dfs -deleteSnapshot /dir s1 > space consumed with Quotausage is 45bytes and Content Summary is 15bytes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15204) TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051560#comment-17051560
]
Íñigo Goiri commented on HDFS-15204:
I mark on a calendar the days where we had no failed unit tests :)
I dream about days with no flaky tests.
> TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
> ---
>
> Key: HDFS-15204
> URL: https://issues.apache.org/jira/browse/HDFS-15204
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 2.10.1
>
> Attachments: HDFS-15204-branch-2.10.001.patch, HDFS-15204.001.patch
>
>
> {code:bash}
> testRemoveCacheDescriptor:
> java.lang.AssertionError: CacheHit: 0, 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1344)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testRemoveCacheDescriptor(TestRetryCacheWithHA.java:1245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15204) TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15204:
---
Fix Version/s: 2.10.1
> TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
> ---
>
> Key: HDFS-15204
> URL: https://issues.apache.org/jira/browse/HDFS-15204
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 2.10.1
>
> Attachments: HDFS-15204-branch-2.10.001.patch, HDFS-15204.001.patch
>
>
> {code:bash}
> testRemoveCacheDescriptor:
> java.lang.AssertionError: CacheHit: 0, 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1344)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testRemoveCacheDescriptor(TestRetryCacheWithHA.java:1245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15204) TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051484#comment-17051484
]
Íñigo Goiri commented on HDFS-15204:
Thanks [~ahussein] for the fix.
Committed to trunk, branch-3.2 and branch-2.10.
It did not apply to branch-3.1 and back.
> TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
> ---
>
> Key: HDFS-15204
> URL: https://issues.apache.org/jira/browse/HDFS-15204
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 2.10.1
>
> Attachments: HDFS-15204-branch-2.10.001.patch, HDFS-15204.001.patch
>
>
> {code:bash}
> testRemoveCacheDescriptor:
> java.lang.AssertionError: CacheHit: 0, 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1344)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testRemoveCacheDescriptor(TestRetryCacheWithHA.java:1245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15204) TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15204:
---
Fix Version/s: 3.2.2
3.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
> ---
>
> Key: HDFS-15204
> URL: https://issues.apache.org/jira/browse/HDFS-15204
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0, 3.2.2
>
> Attachments: HDFS-15204-branch-2.10.001.patch, HDFS-15204.001.patch
>
>
> {code:bash}
> testRemoveCacheDescriptor:
> java.lang.AssertionError: CacheHit: 0, 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1344)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testRemoveCacheDescriptor(TestRetryCacheWithHA.java:1245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10498) Intermittent test failure org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength.testSnapshotfileLength
[
https://issues.apache.org/jira/browse/HDFS-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein reassigned HDFS-10498:
Assignee: Ahmed Hussein
> Intermittent test failure
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength.testSnapshotfileLength
> ---
>
> Key: HDFS-10498
> URL: https://issues.apache.org/jira/browse/HDFS-10498
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Hanisha Koneru
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: test_failure.txt
>
>
> Error Details
> Per https://builds.apache.org/job/PreCommit-HDFS-Build/15646/testReport/, we
> had the following failure. Local rerun is successful.
> Error Details:
> {panel}
> Fail to get block MD5 for
> LocatedBlock{BP-145245805-172.17.0.3-1464981728847:blk_1073741826_1002;
> getBlockSize()=1; corrupt=false; offset=1024;
> locs=[DatanodeInfoWithStorage[127.0.0.1:55764,DS-a33d7c97-9d4a-4694-a47e-a3187a33ed5a,DISK]]}
> {panel}
> Stack Trace:
> {panel}
> java.io.IOException: Fail to get block MD5 for
> LocatedBlock{BP-145245805-172.17.0.3-1464981728847:blk_1073741826_1002;
> getBlockSize()=1; corrupt=false; offset=1024;
> locs=[DatanodeInfoWithStorage[127.0.0.1:55764,DS-a33d7c97-9d4a-4694-a47e-a3187a33ed5a,DISK]]}
> at
> org.apache.hadoop.hdfs.FileChecksumHelper$ReplicatedFileChecksumComputer.checksumBlocks(FileChecksumHelper.java:289)
> at
> org.apache.hadoop.hdfs.FileChecksumHelper$FileChecksumComputer.compute(FileChecksumHelper.java:206)
> at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:1731)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$31.doCall(DistributedFileSystem.java:1482)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$31.doCall(DistributedFileSystem.java:1479)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1490)
> at
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength.testSnapshotfileLength(TestSnapshotFileLength.java:137)
> Standard Output 7 sec
> {panel}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10961) Flaky test TestSnapshotFileLength.testSnapshotfileLength
[
https://issues.apache.org/jira/browse/HDFS-10961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein reassigned HDFS-10961:
Assignee: Ahmed Hussein
> Flaky test TestSnapshotFileLength.testSnapshotfileLength
>
>
> Key: HDFS-10961
> URL: https://issues.apache.org/jira/browse/HDFS-10961
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client
>Affects Versions: 3.0.0-alpha1
>Reporter: Yongjun Zhang
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: flaky-test
>
> Flaky test TestSnapshotFileLength.testSnapshotfileLength
> {code}
> Error Message
> Unable to close file because the last block does not have enough number of
> replicas.
> Stack Trace
> java.io.IOException: Unable to close file because the last block does not
> have enough number of replicas.
> at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2630)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2592)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2546)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
> at
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength.testSnapshotfileLength(TestSnapshotFileLength.java:130)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15204) TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051523#comment-17051523
]
Ahmed Hussein commented on HDFS-15204:
--
Thanks [~inigoiri] for reviewing and committing the patches. We are on fire
killing those bugs :)
> TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently
> ---
>
> Key: HDFS-15204
> URL: https://issues.apache.org/jira/browse/HDFS-15204
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 2.10.1
>
> Attachments: HDFS-15204-branch-2.10.001.patch, HDFS-15204.001.patch
>
>
> {code:bash}
> testRemoveCacheDescriptor:
> java.lang.AssertionError: CacheHit: 0, 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1344)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testRemoveCacheDescriptor(TestRetryCacheWithHA.java:1245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6874) Add GETFILEBLOCKLOCATIONS operation to HttpFS
[
https://issues.apache.org/jira/browse/HDFS-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051543#comment-17051543
]
hemanthboyina commented on HDFS-6874:
-
In webhdfs :
when client call getfileblocklocations , webhdfs calls client protocol's
get_block_locations and NN returns with LocatedBlocks and webhdfs converts
Locatedblocks to BlockLocations[]
In httpfs :
when client call getfileblocklocations , httpfs calls file system api
getfileblocklocations which retuns blocklocations[]
But In Httpfs with Webhdfs
when client calls getfileblocklocations , httpfs return blocklocations[] but
webhdfs expects Locatedblocks , so there is an issue here
any suggestions [~elgoiri] [~kihwal]
please correct me if i am missing something here
> Add GETFILEBLOCKLOCATIONS operation to HttpFS
> -
>
> Key: HDFS-6874
> URL: https://issues.apache.org/jira/browse/HDFS-6874
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Affects Versions: 2.4.1, 2.7.3
>Reporter: Gao Zhong Liang
>Assignee: Weiwei Yang
>Priority: Major
> Labels: BB2015-05-TBR
> Attachments: HDFS-6874-1.patch, HDFS-6874-branch-2.6.0.patch,
> HDFS-6874.02.patch, HDFS-6874.03.patch, HDFS-6874.04.patch,
> HDFS-6874.05.patch, HDFS-6874.06.patch, HDFS-6874.07.patch,
> HDFS-6874.08.patch, HDFS-6874.09.patch, HDFS-6874.10.patch, HDFS-6874.patch
>
>
> GETFILEBLOCKLOCATIONS operation is missing in HttpFS, which is already
> supported in WebHDFS. For the request of GETFILEBLOCKLOCATIONS in
> org.apache.hadoop.fs.http.server.HttpFSServer, BAD_REQUEST is returned so far:
> ...
> case GETFILEBLOCKLOCATIONS: {
> response = Response.status(Response.Status.BAD_REQUEST).build();
> break;
> }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14977) Quota Usage and Content summary are not same in Truncate with Snapshot
[ https://issues.apache.org/jira/browse/HDFS-14977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051491#comment-17051491 ] Íñigo Goiri commented on HDFS-14977: Thanks [~hemanthboyina] for the patch and [~surendrasingh] for the review. Committed to trunk. > Quota Usage and Content summary are not same in Truncate with Snapshot > --- > > Key: HDFS-14977 > URL: https://issues.apache.org/jira/browse/HDFS-14977 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14977.001.patch, HDFS-14977.002.patch, > HDFS-14977.003.patch > > > steps : hdfs dfs -mkdir /dir > hdfs dfs -put file /dir (file size = 10bytes) > hdfs dfsadmin -allowSnapshot /dir > hdfs dfs -createSnapshot /dir s1 > space consumed with Quotausage and Content Summary is 30bytes > hdfs dfs -truncate -w 5 /dir/file > space consumed with Quotausage , Content Summary is 45 bytes > hdfs dfs -deleteSnapshot /dir s1 > space consumed with Quotausage is 45bytes and Content Summary is 15bytes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15193) Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`
[
https://issues.apache.org/jira/browse/HDFS-15193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ctest updated HDFS-15193:
-
Attachment: HDFS-15193-003.patch
> Improving the error message for missing
> `dfs.namenode.rpc-address.$NAMESERVICE`
> ---
>
> Key: HDFS-15193
> URL: https://issues.apache.org/jira/browse/HDFS-15193
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ctest
>Assignee: Ctest
>Priority: Minor
> Attachments: HDFS-15193-001.patch, HDFS-15193-002.patch,
> HDFS-15193-003.patch
>
>
> I set `dfs.nameservices` with the value of one name service (let’s say `ns1`
> for simplicity) and forgot to set `dfs.namenode.rpc-address.ns1`.
> Then I got an error message:
> {code:java}
> [ERROR] testNonFederation(org.apache.hadoop.hdfs.tools.TestGetConf) Time
> elapsed: 0.195 s <<< ERROR!
> java.io.IOException: Incorrect configuration: namenode address
> dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
> at
> org.apache.hadoop.hdfs.DFSUtil.getNNServiceRpcAddressesForCluster(DFSUtil.java:629)
> at
> org.apache.hadoop.hdfs.tools.TestGetConf.getAddressListFromConf(TestGetConf.java:132)
> at
> org.apache.hadoop.hdfs.tools.TestGetConf.verifyAddresses(TestGetConf.java:234)
> at
> org.apache.hadoop.hdfs.tools.TestGetConf.testNonFederation(TestGetConf.java:308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> The error message above told me that `dfs.namenode.rpc-address` or
> `dfs.namenode.servicerpc-address` should be set.
> However, the actual reason for the error is that
> `dfs.namenode.rpc-address.ns1` or `dfs.namenode.servicerpc-address.ns1` is
> not set.
>
> *How to improve*
> I wrote a patch to improve the error message. Here is the current error
> message:
> {code:java}
> [ERROR] testNonFederation(org.apache.hadoop.hdfs.tools.TestGetConf) Time
> elapsed: 0.195 s <<< ERROR!
> java.io.IOException: Incorrect configuration: namenode address
> dfs.namenode.servicerpc-address[.$NAMESERVICE] or
> dfs.namenode.rpc-address[.$NAMESERVICE] is not configured.
> {code}
> Then the users can immediately know that they should set
> `dfs.namenode.rpc-address.ns1` or `dfs.namenode.servicerpc-address.ns1`
> according to the error message.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15196) RBF: RouterRpcServer getListing cannot list large dirs correctly
[
https://issues.apache.org/jira/browse/HDFS-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051493#comment-17051493
]
Íñigo Goiri commented on HDFS-15196:
Thanks [~fengnanli] for the clarification.
Yes, I agree we should go with #1 and guarrantee the order in the Router and
not the client.
{quote}
We may want to also test the limit in the Router side.
{quote}
I was referring to the Router returning batches.
In general I agree with [~LiJinglun], we should keep mount points as folders
and we should add both tests.
> RBF: RouterRpcServer getListing cannot list large dirs correctly
>
>
> Key: HDFS-15196
> URL: https://issues.apache.org/jira/browse/HDFS-15196
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Critical
> Attachments: HDFS-15196.001.patch, HDFS-15196.002.patch,
> HDFS-15196.003.patch, HDFS-15196.003.patch, HDFS-15196.004.patch,
> HDFS-15196.005.patch
>
>
> In RouterRpcServer, getListing function is handled as two parts:
> # Union all partial listings from destination ns + paths
> # Append mount points for the dir to be listed
> In the case of large dir which is bigger than DFSConfigKeys.DFS_LIST_LIMIT
> (with default value 1k), the batch listing will be used and the startAfter
> will be used to define the boundary of each batch listing. However, step 2
> here will add existing mount points, which will mess up with the boundary of
> the batch, thus making the next batch startAfter wrong.
> The fix is just to append the mount points when there is no more batch query
> necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15193) Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`
[
https://issues.apache.org/jira/browse/HDFS-15193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051586#comment-17051586
]
Hadoop QA commented on HDFS-15193:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 38s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.TestEncryptionZones |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.7 Server=19.03.7 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15193 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12995642/HDFS-15193-003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4b7727348ee2 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bbd704b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28896/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28896/testReport/ |
| Max. process+thread count | 3269 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-15193) Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`
[
https://issues.apache.org/jira/browse/HDFS-15193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051661#comment-17051661
]
Ctest commented on HDFS-15193:
--
[~ayushtkn]
Thank you for your help!
I uploaded a new patch (003) containing a unit test for the change in error msg.
> Improving the error message for missing
> `dfs.namenode.rpc-address.$NAMESERVICE`
> ---
>
> Key: HDFS-15193
> URL: https://issues.apache.org/jira/browse/HDFS-15193
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Ctest
>Assignee: Ctest
>Priority: Minor
> Attachments: HDFS-15193-001.patch, HDFS-15193-002.patch,
> HDFS-15193-003.patch
>
>
> I set `dfs.nameservices` with the value of one name service (let’s say `ns1`
> for simplicity) and forgot to set `dfs.namenode.rpc-address.ns1`.
> Then I got an error message:
> {code:java}
> [ERROR] testNonFederation(org.apache.hadoop.hdfs.tools.TestGetConf) Time
> elapsed: 0.195 s <<< ERROR!
> java.io.IOException: Incorrect configuration: namenode address
> dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
> at
> org.apache.hadoop.hdfs.DFSUtil.getNNServiceRpcAddressesForCluster(DFSUtil.java:629)
> at
> org.apache.hadoop.hdfs.tools.TestGetConf.getAddressListFromConf(TestGetConf.java:132)
> at
> org.apache.hadoop.hdfs.tools.TestGetConf.verifyAddresses(TestGetConf.java:234)
> at
> org.apache.hadoop.hdfs.tools.TestGetConf.testNonFederation(TestGetConf.java:308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> The error message above told me that `dfs.namenode.rpc-address` or
> `dfs.namenode.servicerpc-address` should be set.
> However, the actual reason for the error is that
> `dfs.namenode.rpc-address.ns1` or `dfs.namenode.servicerpc-address.ns1` is
> not set.
>
> *How to improve*
> I wrote a patch to improve the error message. Here is the current error
> message:
> {code:java}
> [ERROR] testNonFederation(org.apache.hadoop.hdfs.tools.TestGetConf) Time
> elapsed: 0.195 s <<< ERROR!
> java.io.IOException: Incorrect configuration: namenode address
> dfs.namenode.servicerpc-address[.$NAMESERVICE] or
> dfs.namenode.rpc-address[.$NAMESERVICE] is not configured.
> {code}
> Then the users can immediately know that they should set
> `dfs.namenode.rpc-address.ns1` or `dfs.namenode.servicerpc-address.ns1`
> according to the error message.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14442) Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId
[
https://issues.apache.org/jira/browse/HDFS-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051819#comment-17051819
]
Surendra Singh Lilhore commented on HDFS-14442:
---
+1
{quote}v003 patch LGTM, I will commit once I get a chance to verify the tests
locally.
{quote}
[~xkrogen], any comment ?
> Disagreement between HAUtil.getAddressOfActive and
> RpcInvocationHandler.getConnectionId
> ---
>
> Key: HDFS-14442
> URL: https://issues.apache.org/jira/browse/HDFS-14442
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Ravuri Sushma sree
>Priority: Major
> Attachments: HDFS-14442.001.patch, HDFS-14442.002.patch,
> HDFS-14442.003.patch, HDFS-14442.004.patch
>
>
> While working on HDFS-14245, we noticed a discrepancy in some proxy-handling
> code.
> The description of {{RpcInvocationHandler.getConnectionId()}} states:
> {code}
> /**
>* Returns the connection id associated with the InvocationHandler instance.
>* @return ConnectionId
>*/
> ConnectionId getConnectionId();
> {code}
> It does not make any claims about whether this connection ID will be an
> active proxy or not. Yet in {{HAUtil}} we have:
> {code}
> /**
>* Get the internet address of the currently-active NN. This should rarely
> be
>* used, since callers of this method who connect directly to the NN using
> the
>* resulting InetSocketAddress will not be able to connect to the active NN
> if
>* a failover were to occur after this method has been called.
>*
>* @param fs the file system to get the active address of.
>* @return the internet address of the currently-active NN.
>* @throws IOException if an error occurs while resolving the active NN.
>*/
> public static InetSocketAddress getAddressOfActive(FileSystem fs)
> throws IOException {
> if (!(fs instanceof DistributedFileSystem)) {
> throw new IllegalArgumentException("FileSystem " + fs + " is not a
> DFS.");
> }
> // force client address resolution.
> fs.exists(new Path("/"));
> DistributedFileSystem dfs = (DistributedFileSystem) fs;
> DFSClient dfsClient = dfs.getClient();
> return RPC.getServerAddress(dfsClient.getNamenode());
> }
> {code}
> Where the call {{RPC.getServerAddress()}} eventually terminates into
> {{RpcInvocationHandler#getConnectionId()}}, via {{RPC.getServerAddress()}} ->
> {{RPC.getConnectionIdForProxy()}} ->
> {{RpcInvocationHandler#getConnectionId()}}. {{HAUtil}} appears to be making
> an incorrect assumption that {{RpcInvocationHandler}} will necessarily return
> an _active_ connection ID. {{ObserverReadProxyProvider}} demonstrates a
> counter-example to this, since the current connection ID may be pointing at,
> for example, an Observer NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15120) Refresh BlockPlacementPolicy at runtime.
[ https://issues.apache.org/jira/browse/HDFS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051825#comment-17051825 ] Ayush Saxena commented on HDFS-15120: - Hi [~LiJinglun] Observed one problem here, this fix will work if we want to change the value back to default. but not a new one. since you are not setting the new value in the conf. In the test also we just tested from MockBPP to DefaultBPP. and this works. But I think the reverse won't work. Give a check once > Refresh BlockPlacementPolicy at runtime. > > > Key: HDFS-15120 > URL: https://issues.apache.org/jira/browse/HDFS-15120 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15120.001.patch, HDFS-15120.002.patch, > HDFS-15120.003.patch, HDFS-15120.004.patch, HDFS-15120.005.patch, > HDFS-15120.006.patch, HDFS-15120.007.1.patch, HDFS-15120.007.patch, > HDFS-15120.008.patch > > > Now if we want to switch BlockPlacementPolicies we need to restart the > NameNode. It would be convenient if we can switch it at runtime. For example > we can switch between AvailableSpaceBlockPlacementPolicy and > BlockPlacementPolicyDefault as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15205) FSImage sort section logic is wrong
[
https://issues.apache.org/jira/browse/HDFS-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051017#comment-17051017
]
Stephen O'Donnell commented on HDFS-15205:
--
Hi [~angerszhuuu] With the patch you have uploaded, does this fix the load
order problem you are seeing on branch-2?
Looking at the logic, your change should only have an impact when one of the
section names is null, and I think that should never be the case. Therefore I
am wondering if there is something else going wrong when you backported the
patch to branch 2.6. We have also seen this patch used in a few places and this
has not been reported before.
It does look like this is a bug in the sorting, but it has probably gone
unnoticed as there should never be any null section names.
Can I ask:
1. Did the patch backport cleanly to 2.6 or did you need to make some changes?
2. With and without your fix, could you print out the order of the items in the
"sections" list after sorting it so we can see what it is doing, and also log a
message if any of the sections are null, eg:
{code}
...
ArrayList sections = Lists.newArrayList(summary
.getSectionsList());
Collections.sort(sections, new Comparator() {
@Override
public int compare(FileSummary.Section s1, FileSummary.Section s2) {
SectionName n1 = SectionName.fromString(s1.getName());
SectionName n2 = SectionName.fromString(s2.getName());
if (n1 == null) {
LOG.warn("n1 is null");
return n2 == null ? 0 : -1;
} else if (n2 == null) {
LOG.warn("n2 is null but n1 is not");
return -1;
} else {
return n1.ordinal() - n2.ordinal();
}
}
});
for (FileSummary.Section s : sections) {
log.warn("Section: " + s.getName());
}
{code}
3. If you check the SectionName enum in FSImageFormatProtobuf.java, are all the
sections present in your image also defined in the enum?
> FSImage sort section logic is wrong
> ---
>
> Key: HDFS-15205
> URL: https://issues.apache.org/jira/browse/HDFS-15205
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: angerszhu
>Priority: Blocker
> Attachments: HDFS-15205.001.patch
>
>
> When load FSImage, it will sort sections in FileSummary and load Section's in
> SectionName enum sequence. But the sort method is wrong , when I use
> branch-2.6.0 to load fsimage write by branch-2 with patch
> https://issues.apache.org/jira/browse/HDFS-14771, it will throw NPE because
> it load INODE first
> {code:java}
> 2020-03-03 14:33:26,618 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadPermission(FSImageFormatPBINode.java:101)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectory(FSImageFormatPBINode.java:148)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadRootINode(FSImageFormatPBINode.java:332)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:218)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:254)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:180)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1036)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1020)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:741)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1092)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:780)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:666)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:838)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:817)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)
> {code}
> I print the load
