Decommissioned datanode is counted in service cause datanode allcating failure

2017-11-15 Thread Xie Gang
(); } cacheCapacity -= node.getCacheCapacity(); cacheUsed -= node.getCacheUsed(); } -- Xie Gang

Block invalid IOException causes the DFSClient domain socket being disabled

2017-10-25 Thread Xie Gang
rt-circuit access. " + resp.getMessage()); clientContext.getDomainSocketFactory() .disableShortCircuitForPath(pathInfo.getPath()); return null; } -- Xie Gang

Re: Block invalid IOException causes the DFSClient domain socket being disabled

2017-10-26 Thread Xie Gang
Shall I create the jira directly? On Thu, Oct 26, 2017 at 12:34 PM, Xie Gang <xiegang...@gmail.com> wrote: > Hi, > > We use HDFS2.4 & 2.6, and recently hit a issue that DFSClient domain > socket is disabled when datanode throw block invalid exception. > > The block is

Re: Inconsistence between the datanode volume info and OS df

2018-01-15 Thread Xie Gang
Got the root cause, it's a dup of HDFS-8072 https://issues.apache.org/jira/browse/HDFS-8072 On Wed, Jan 10, 2018 at 2:20 PM, Xie Gang <xiegang...@gmail.com> wrote: > Hi, > > Recently, we hit an issue that, there is a difference between the > freeSpace of the datanode volume

Inconsistence between the datanode volume info and OS df

2018-01-09 Thread Xie Gang
ain. The yarn shared the same server of the dn and has some file cache. Could it be related? The direct cause is that the freeSpace from dn is quit different from the available space from df. After tracking down the code, freeSpace of the dn is from dirFile.getUsableSpace(). could it have some p

Re: does it make sense to get remaining space by sum all the ones of the datanode

2018-01-29 Thread Xie Gang
kum...@apache.org> wrote: > in which version of Hadoop you are seeing this? > > -Vinay > > On 29 Jan 2018 3:26 pm, "Xie Gang" <xiegang...@gmail.com> wrote: > > Hello, > > We recently hit a issue that almost all the disk of the datanode got full > even we

does it make sense to get remaining space by sum all the ones of the datanode

2018-01-29 Thread Xie Gang
ize) { logNodeIsNotChosen(storage, "the node does not have enough " + storage.getStorageType() + " space" + " (required=" + requiredSize + ", scheduled=" + scheduledSize + ", remaining=" + remaining + ")"); stats.incrOverScheduled(); return false; } -- Xie Gang

enable the SC local read to UC block to optimize the read perf

2018-02-01 Thread Xie Gang
, and will look into it further. But not sure if we tried this before. -- Xie Gang

Why to set socket read timeout to n*socketTimeout in data transfer

2018-02-04 Thread Xie Gang
argets.length * dnConf.socketTimeout);<<<<-* long writeTimeout = dnConf.socketWriteTimeout + HdfsServerConstants.WRITE_TIMEOUT_EXTENSION * (targets.length-1); -- Xie Gang

Why always allocate shm slot when local read even if no zero copy needed?

2018-02-08 Thread Xie Gang
ShortCircuitReplica(ExtendedBlockId key, FileInputStream dataStream, FileInputStream metaStream, ShortCircuitCache cache, long creationTimeMs, Slot slot) throws IOException { -- Xie Gang

How to access 2 HDFS with difference version in one app

2017-12-28 Thread Xie Gang
, is there any other way to do this? The rough idea is to change the RPC engine to change the shaded package name back to the original one. but not sure if it could work. -- Xie Gang