[jira] [Created] (HDFS-5492) Port HDFS-2069 (Incorrect default trash interval in the docs) to trunk
Akira AJISAKA created HDFS-5492:
---
Summary: Port HDFS-2069 (Incorrect default trash interval in the
docs) to trunk
Key: HDFS-5492
URL: https://issues.apache.org/jira/browse/HDFS-5492
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Priority: Minor
HDFS-2069 is not ported to current document.
The description of HDFS-2069 is as follows:
{quote}
Current HDFS architecture information about Trash is incorrectly documented as -
The current default policy is to delete files from /trash that are more than 6
hours old. In the future, this policy will be configurable through a well
defined interface.
It should be something like -
Current default trash interval is set to 0 (Deletes file without storing in
trash ) . This value is configurable parameter stored as fs.trash.interval
stored in core-site.xml .
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
Hadoop-Hdfs-0.23-Build - Build # 788 - Still Failing
See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/788/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7885 lines...] [ERROR] location: class com.google.protobuf.InvalidProtocolBufferException [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3313,27] cannot find symbol [ERROR] symbol : method setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto) [ERROR] location: class com.google.protobuf.InvalidProtocolBufferException [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3319,8] cannot find symbol [ERROR] symbol : method makeExtensionsImmutable() [ERROR] location: class org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3330,10] cannot find symbol [ERROR] symbol : method ensureFieldAccessorsInitialized(java.lang.Classorg.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto,java.lang.Classorg.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto.Builder) [ERROR] location: class com.google.protobuf.GeneratedMessage.FieldAccessorTable [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3335,31] cannot find symbol [ERROR] symbol : class AbstractParser [ERROR] location: package com.google.protobuf [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3344,4] method does not override or implement a method from a supertype [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[4098,12] cannot find symbol [ERROR] symbol : method ensureFieldAccessorsInitialized(java.lang.Classorg.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto,java.lang.Classorg.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto.Builder) [ERROR] location: class com.google.protobuf.GeneratedMessage.FieldAccessorTable [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[4371,104] cannot find symbol [ERROR] symbol : method getUnfinishedMessage() [ERROR] location: class com.google.protobuf.InvalidProtocolBufferException [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5264,8] getUnknownFields() in org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto cannot override getUnknownFields() in com.google.protobuf.GeneratedMessage; overridden method is final [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5284,19] cannot find symbol [ERROR] symbol : method parseUnknownField(com.google.protobuf.CodedInputStream,com.google.protobuf.UnknownFieldSet.Builder,com.google.protobuf.ExtensionRegistryLite,int) [ERROR] location: class org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5314,15] cannot find symbol [ERROR] symbol : method setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto) [ERROR] location: class com.google.protobuf.InvalidProtocolBufferException [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5317,27] cannot find symbol [ERROR] symbol : method setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto) [ERROR] location: class com.google.protobuf.InvalidProtocolBufferException [ERROR]
Build failed in Jenkins: Hadoop-Hdfs-0.23-Build #788
See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/788/ -- [...truncated 7692 lines...] [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[270,37] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[281,30] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[10533,37] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[10544,30] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[8357,37] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[8368,30] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[12641,37] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[12652,30] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[9741,37] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[9752,30] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[1781,37] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[1792,30] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5338,37] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5349,30] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[6290,37] cannot find symbol [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR] https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/ws/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[6301,30] cannot find symbol [ERROR] symbol : class
Re: Next releases
Hi Arun, Another feature that would be relevant and got deferred was the symlink work (HADOOP-10020) that Colin and Andrew were working on. Can we include this in hadoop-2.3.0 also? thanks hari On Sun, Nov 10, 2013 at 2:07 PM, Alejandro Abdelnur t...@cloudera.comwrote: Arun, thanks for jumping on this. On hadoop branch-2.2. I've quickly scanned the commit logs starting from the 2.2.0 release and I've found around 20 JIRAs that I like seeing in 2.2.1. Not all of them are bugs but the don't shake anything and improve usability. I presume others will have their own laundry lists as well and I wonder the union of all of them how much adds up to the current 81 commits. How about splitting the JIRAs among a few contributors to assert there is nothing risky in there? And if so get discuss getting rid of those commits for 2.2.1. IMO doing that would be cheaper than selectively applying commits on a fresh branch. Said this, I think we should get 2.2.1 out of the door before switching main efforts to 2.3.0. I volunteer myself to drive 2.2.1 a release if ASAP if you don't have the bandwidth at the moment for it. Cheers. Alejandro Commits in branch-2.2 that I'd like them to be in the 2.2.1 release: The ones prefixed with '*' technically are not bugs. YARN-1284. LCE: Race condition leaves dangling cgroups entries for killed containers. (Alejandro Abdelnur via Sandy Ryza) YARN-1265. Fair Scheduler chokes on unhealthy node reconnect (Sandy Ryza) YARN-1044. used/min/max resources do not display info in the scheduler page (Sangjin Lee via Sandy Ryza) YARN-305. Fair scheduler logs too many Node offered to app messages. (Lohit Vijayarenu via Sandy Ryza) *MAPREDUCE-5463. Deprecate SLOTS_MILLIS counters. (Tzuyoshi Ozawa via Sandy Ryza) YARN-1259. In Fair Scheduler web UI, queue num pending and num active apps switched. (Robert Kanter via Sandy Ryza) YARN-1295. In UnixLocalWrapperScriptBuilder, using bash -c can cause Text file busy errors. (Sandy Ryza) *MAPREDUCE-5457. Add a KeyOnlyTextOutputReader to enable streaming to write out text files without separators (Sandy Ryza) *YARN-1258. Allow configuring the Fair Scheduler root queue (Sandy Ryza) *YARN-1288. Make Fair Scheduler ACLs more user friendly (Sandy Ryza) YARN-1330. Fair Scheduler: defaultQueueSchedulingPolicy does not take effect (Sandy Ryza) HDFS-5403. WebHdfs client cannot communicate with older WebHdfs servers post HDFS-5306. Contributed by Aaron T. Myers. *YARN-1335. Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication (Sandy Ryza) *YARN-1333. Support blacklisting in the Fair Scheduler (Tsuyoshi Ozawa via Sandy Ryza) *MAPREDUCE-4680. Job history cleaner should only check timestamps of files in old enough directories (Robert Kanter via Sandy Ryza) YARN-1109. Demote NodeManager Sending out status for container logs to debug (haosdent via Sandy Ryza) *YARN-1321. Changed NMTokenCache to support both singleton and an instance usage. Contributed by Alejandro Abdelnur YARN-1343. NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs. (tucu) YARN-1381. Same relaxLocality appears twice in exception message of AMRMClientImpl#checkLocalityRelaxationConflict() (Ted Yu via Sandy Ryza) HADOOP-9898. Set SO_KEEPALIVE on all our sockets. Contributed by Todd Lipcon. YARN-1388. Fair Scheduler page always displays blank fair share (Liyin Liang via Sandy Ryza) On Fri, Nov 8, 2013 at 10:35 PM, Chris Nauroth cnaur...@hortonworks.com wrote: Arun, what are your thoughts on test-only patches? I know I've been merging a lot of Windows test stabilization patches down to branch-2.2. These can't rightly be called blockers, but they do improve dev experience, and there is no risk to product code. Chris Nauroth Hortonworks http://hortonworks.com/ On Fri, Nov 8, 2013 at 1:30 AM, Steve Loughran ste...@hortonworks.com wrote: On 8 November 2013 02:42, Arun C Murthy a...@hortonworks.com wrote: Gang, Thinking through the next couple of releases here, appreciate f/b. # hadoop-2.2.1 I was looking through commit logs and there is a *lot* of content here (81 commits as on 11/7). Some are features/improvements and some are fixes - it's really hard to distinguish what is important and what isn't. I propose we start with a blank slate (i.e. blow away branch-2.2 and start fresh from a copy of branch-2.2.0) and then be very careful and meticulous about including only *blocker* fixes in branch-2.2. So, most of the content here comes via the next minor release (i.e. hadoop-2.3) In future, we continue to be *very* parsimonious about what gets into a patch release (major.minor.patch) - in general, these should be only *blocker* fixes or
[jira] [Created] (HDFS-5493) DFSClient#DFSInputStream#blockSeekTo may leak socket connection.
Chris Nauroth created HDFS-5493:
---
Summary: DFSClient#DFSInputStream#blockSeekTo may leak socket
connection.
Key: HDFS-5493
URL: https://issues.apache.org/jira/browse/HDFS-5493
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 1.2.1
Reporter: Chris Nauroth
{{DFSClient#DFSInputStream#blockSeekTo}} may handle {{IOException}} by
refetching a new block access token and then reattempting {{fetchBlockAt}}.
However, {{fetchBlockAt}} may then throw its own {{IOException}}. If this
happens, then the method skips calling {{Socket#close}}. This is likely to
manifest as a leak of sockets left in CLOSE_WAIT status.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
Re: Exception in mid of reading files.
Divya, thank you for reporting back on this. Nicholas and I had an offline
conversation and came to the conclusion that this is likely to be a
different problem from HDFS-3373. Although the symptoms look similar, the
socket caching code mentioned in HDFS-3373 is not present in branch-1.
I filed a new issue for your bug report: HDFS-5493. Nicholas pointed out a
spot in the DFSClient code where we may have a socket leak.
https://issues.apache.org/jira/browse/HDFS-5493
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Thu, Nov 7, 2013 at 12:34 AM, Divya R avyakr...@gmail.com wrote:
Hi Chris,
Thanks a lot for the help. But after lot of investigation I found that
the issue was with the cached socket connection which was raised as a bug
by Nicholas. Bug details are as follows,
HDFS-3373 https://issues.apache.org/jira/browse/HDFS-3373 FileContext
HDFS implementation can leak socket caches
When I executed command netstat -a |grep 50010 the count was approximately
52000. This issue is fixed in
0.20.3https://issues.apache.org/jira/browse/HDFS/fixforversion/12314814,
0.20.205.0
https://issues.apache.org/jira/browse/HDFS/fixforversion/12316392,
but its not present in hadoop-1.2.X. Could you please guide me as to what
could I do.?
-Divya
On Sat, Oct 26, 2013 at 12:38 AM, Chris Nauroth cnaur...@hortonworks.com
wrote:
Hi Divya,
The exceptions indicate that the HDFS client failed to establish a
network
connection to a datanode hosting a block that the client is trying to
read.
After too many of these failures (default 3, but configurable), the HDFS
client aborts the read and this bubbles up to the caller with the could
not obtain block error.
I recommend troubleshooting this as a network connectivity issue. This
wiki page includes a few tips as a starting point:
http://wiki.apache.org/hadoop/TroubleShooting
Hope this helps,
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Fri, Oct 25, 2013 at 4:53 AM, Divya R avyakr...@gmail.com wrote:
Hi Guys,
I'm indexing data (~50 -100GB per day) from hadoop. Hadoop is
Running
in
cluster mode (having 2 dataNaodes currently). After every two or three
hours I'm getting this exception. But both Data nodes are up and
running.
Can any one please guide me as to what I should do or If I'm doing
wrong.
Code Snippet:
public InitHadoop() {
configuration = new Configuration();
configuration.set(fs.default.name, hdfs://namenode
IP:54310); // Is this write to specify on namenode IP.?
configuration.set(mapred.job.tracker, hdfs://namenode
IP:54311);
try {
fileSystem = FileSystem.get(configuration);
} catch (IOException e) {
e.printStackTrace();
}
}
private void indexDocument(FSDataInputStream file) {
Scanner scanner = new Scanner(file);
while (scanner.hasNext() != null) {
// Indexing code
}
}
}
Logs:
2013-10-25 09:37:57 WARN DFSClient:2266 - Failed to connect to
/IP:50010, add to deadNodes and continuejava.net.BindException:
Cannot
assign requested address
2013-10-25 09:37:57 WARN DFSClient:2266 - Failed to connect to
/IP:50010, add to deadNodes and continuejava.net.BindException:
Cannot
assign requested address
2013-10-25 09:37:57 INFO DFSClient:2432 - Could not obtain block
blk_-8795538519317154213_432897 from any node: java.io.IOException: No
live
nodes contain current block. Will get new block locations from namenode
and
retry...
2013-10-25 09:37:58 WARN DFSClient:2266 - Failed to connect to
/IP:50010, add to deadNodes and continuejava.net.BindException:
Cannot
assign requested address
2013-10-25 09:37:58 WARN DFSClient:2266 - Failed to connect to
/IP:50010, add to deadNodes and continuejava.net.BindException:
Cannot
assign requested address
2013-10-25 09:37:58 INFO DFSClient:2432 - Could not obtain block
blk_-5974673190155585497_432671 from any node: java.io.IOException: No
live
nodes contain current block. Will get new block locations from namenode
and
retry...
2013-10-25 09:37:59 WARN DFSClient:2266 - Failed to connect to
/IP:50010, add to deadNodes and continuejava.net.BindException:
Cannot
assign requested address
2013-10-25 09:37:59 WARN DFSClient:2266 - Failed to connect to
/IP:50010, add to deadNodes and continuejava.net.BindException:
Cannot
assign requested address
2013-10-25 09:37:59 INFO DFSClient:2432 - Could not obtain block
blk_-1662761320365439855_431653 from any node: java.io.IOException: No
live
nodes contain current block. Will get new block locations from namenode
and
retry...
2013-10-25 09:37:59 WARN DFSClient:2266 - Failed to connect to
/IP:50010, add to deadNodes and continuejava.net.BindException:
Cannot
[jira] [Created] (HDFS-5494) Fix findbugs warnings for HDFS-2832
Arpit Agarwal created HDFS-5494: --- Summary: Fix findbugs warnings for HDFS-2832 Key: HDFS-5494 URL: https://issues.apache.org/jira/browse/HDFS-5494 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Arpit Agarwal Assignee: Arpit Agarwal Jenkins flagged the following findbugs warnings for the HDFS-2832 merge patch. https://builds.apache.org/job/PreCommit-HDFS-Build/5384//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5495) Remove further JUnit3 usages from HDFS
Andrew Wang created HDFS-5495:
-
Summary: Remove further JUnit3 usages from HDFS
Key: HDFS-5495
URL: https://issues.apache.org/jira/browse/HDFS-5495
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.2.0, 3.0.0
Reporter: Andrew Wang
We're trying to move away from junit3 in hadoop to junit4. One easy way of
testing for this is something like the following:
{code}
- % ack junit.framework -l
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientExcludedNodes.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileInputStreamCache.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestMultipleNNDataBlockScanner.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestAvailableSpaceVolumeChoosingPolicy.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestXMLUtils.java
hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestURLConnectionFactory.java
hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSKerberosAuthenticationHandler.java
hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java
hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/lib/service/security/TestDelegationTokenManagerService.java
hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java
{code}
Most of these are probably just static assert imports, it's fairly mechanical
to use the JUnit4 versions. Same goes for converting the unit tests (if
necessary).
See HDFS-3583 and HDFS-3711 for the history.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
HDFS read/write data throttling
Hello Devs, Wanted to reach out and see if anyone has thought about ability to throttle data transfer within HDFS. One option we have been thinking is to throttle on a per FileSystem basis, similar to Statistics in FileSystem. This would mean anyone with handle to HDFS/Hftp will be throttled globally within JVM. Right value to come up for this would be based on type of hardware we use and how many tasks/clients we allow. On the other hand doing something like this at FileSystem layer would mean many other tasks such as Job jar copy, DistributedCache copy and any hidden data movement would also be throttled. We wanted to know if anyone has had such requirement on their clusters in the past and what was the thinking around it. Appreciate your inputs/comments -- Have a Nice Day! Lohit
[jira] [Created] (HDFS-5496) Make replication queue initialization asynchronous
Kihwal Lee created HDFS-5496: Summary: Make replication queue initialization asynchronous Key: HDFS-5496 URL: https://issues.apache.org/jira/browse/HDFS-5496 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Kihwal Lee Today, initialization of replication queues blocks safe mode exit and certain HA state transitions. For a big name space, this can take hundreds of seconds with the FSNamesystem write lock held. During this time, important requests (e.g. initial block reports, heartbeat, etc) are blocked. The effect of delaying the initialization would be not starting replication right away, but I think the benefit outweighs. If we make it asynchronous, the work per iteration should be limited, so that the lock duration is capped. If full/incremental block reports and any other requests that modifies block state properly performs replication checks while the blocks are scanned and the queues populated in background, every block will be processed. (Some may be done twice) The replication monitor should run even before all blocks are processed. This will allow namenode to exit safe mode and start serving immediately even with a big name space. It will also reduce the HA failover latency. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: HDFS read/write data throttling
See https://issues.apache.org/jira/browse/HDFS-3475 Please note that this has met with many unexpected impacts on workload. Be careful and be mindful of your Datanode memory and network capacity. On Mon, Nov 11, 2013 at 1:59 PM, lohit lohit.vijayar...@gmail.com wrote: Hello Devs, Wanted to reach out and see if anyone has thought about ability to throttle data transfer within HDFS. One option we have been thinking is to throttle on a per FileSystem basis, similar to Statistics in FileSystem. This would mean anyone with handle to HDFS/Hftp will be throttled globally within JVM. Right value to come up for this would be based on type of hardware we use and how many tasks/clients we allow. On the other hand doing something like this at FileSystem layer would mean many other tasks such as Job jar copy, DistributedCache copy and any hidden data movement would also be throttled. We wanted to know if anyone has had such requirement on their clusters in the past and what was the thinking around it. Appreciate your inputs/comments -- Have a Nice Day! Lohit -- * Adam Muise * Solutions Engineer -- Phone:416-417-4037 Email: amu...@hortonworks.com Website: http://www.hortonworks.com/ * Follow Us: * http://facebook.com/hortonworks/?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature http://twitter.com/hortonworks?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature http://www.linkedin.com/company/hortonworks?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature [image: photo] Latest From Our Blog: How to use R and other non-Java languages in MapReduce and Hive http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HDFS-5497) Performance may suffer when UnderReplicatedBlocks is used heavily
Kihwal Lee created HDFS-5497: Summary: Performance may suffer when UnderReplicatedBlocks is used heavily Key: HDFS-5497 URL: https://issues.apache.org/jira/browse/HDFS-5497 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Kihwal Lee Currently UnderReplicatedBlocks uses LightWeightLinkedSet with the default initial size of 16. If there are a lot of under-replicated blocks, insertion and removal can be very expensive. We see 450K to 1M under-replicated block during start-up, which typically go away soon as last few data nodes join. With 450K under-replicated blocks, replication queue initialization would re-allocate the underlying array 15 time and reinsert elements over 1M times. As block reports come in, it will go through the reverse. I think this one of the reasons why initial block reports after leaving safe mode can take very long time to process. With a larger initial/minimum size, the timing gets significantly shorter. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: HDFS read/write data throttling
Hi Adam, Thanks for the reply. The changes I was referring was in FileSystem.java layer which should not affect HDFS Replication/NameNode operations. To give better idea this would affect clients something like this Configuration conf = new Configuration(); conf.setInt(read.bandwitdh.mbpersec, 20); // 20MB/s FileSystem fs = FileSystem.get(conf); FSDataInputStream fis = fs.open(/path/to/file.xt); fis.read(); // -- This would be max of 20MB/s 2013/11/11 Adam Muise amu...@hortonworks.com See https://issues.apache.org/jira/browse/HDFS-3475 Please note that this has met with many unexpected impacts on workload. Be careful and be mindful of your Datanode memory and network capacity. On Mon, Nov 11, 2013 at 1:59 PM, lohit lohit.vijayar...@gmail.com wrote: Hello Devs, Wanted to reach out and see if anyone has thought about ability to throttle data transfer within HDFS. One option we have been thinking is to throttle on a per FileSystem basis, similar to Statistics in FileSystem. This would mean anyone with handle to HDFS/Hftp will be throttled globally within JVM. Right value to come up for this would be based on type of hardware we use and how many tasks/clients we allow. On the other hand doing something like this at FileSystem layer would mean many other tasks such as Job jar copy, DistributedCache copy and any hidden data movement would also be throttled. We wanted to know if anyone has had such requirement on their clusters in the past and what was the thinking around it. Appreciate your inputs/comments -- Have a Nice Day! Lohit -- * Adam Muise * Solutions Engineer -- Phone:416-417-4037 Email: amu...@hortonworks.com Website: http://www.hortonworks.com/ * Follow Us: * http://facebook.com/hortonworks/?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature http://twitter.com/hortonworks?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature http://www.linkedin.com/company/hortonworks?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature [image: photo] Latest From Our Blog: How to use R and other non-Java languages in MapReduce and Hive http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Have a Nice Day! Lohit
[jira] [Created] (HDFS-5498) Improve datanode startup time
Kihwal Lee created HDFS-5498: Summary: Improve datanode startup time Key: HDFS-5498 URL: https://issues.apache.org/jira/browse/HDFS-5498 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Similarly to HDFS-5027, an improvement can be made for getVomeMap(). This is the phase in which ReplicaMap.is populated. But it will be even better if datanode scans only once and do both. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5499) Provide way to throttle per FileSystem read/write bandwidth
Lohit Vijayarenu created HDFS-5499: -- Summary: Provide way to throttle per FileSystem read/write bandwidth Key: HDFS-5499 URL: https://issues.apache.org/jira/browse/HDFS-5499 Project: Hadoop HDFS Issue Type: Improvement Reporter: Lohit Vijayarenu In some cases it might be worth to throttle read/writer bandwidth on per JVM basis so that clients do not spawn too many thread and start data movement causing other JVMs to starve. Ability to throttle read/write bandwidth on per FileSystem would help avoid such issues. Challenge seems to be how well this can be fit into FileSystem code. If one enables throttling around FileSystem APIs, then any hidden data transfer within cluster using them might also be affected. Eg. copying job jar during job submission, localizing resources for distributed cache and such. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5500) Critical datanode threads may terminate silently on uncaught exceptions
Kihwal Lee created HDFS-5500: Summary: Critical datanode threads may terminate silently on uncaught exceptions Key: HDFS-5500 URL: https://issues.apache.org/jira/browse/HDFS-5500 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Priority: Critical We've seen refreshUsed (DU) thread disappearing on uncaught exceptions. This can go unnoticed for a long time. If OOM occurs, more things can go wrong. On one occasion, Timer, multiple refreshUsed and DataXceiverServer thread had terminated. DataXceiverServer catches OutOfMemoryError and sleeps for 30 seconds, but I am not sure it is really helpful. In once case, the thread did it multiple times then terminated. I suspect another OOM was thrown while in a catch block. As a result, the server socket was not closed and clients hung on connect. If it had at least closed the socket, client-side would have been impacted less. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Next releases
HADOOP-10020 is a JIRA that disables symlinks temporarily. They will be disabled in 2.2.1 as well, if the plan is to have only minor fixes in that branch. To be honest, I'm not aware of anything in 2.2.1 that shouldn't be there. However, I have only been following the HDFS and common side of things so I may not have the full picture. Arun, can you give a specific example of something you'd like to blow away? Colin On Mon, Nov 11, 2013 at 10:06 AM, Hari Mankude hmank...@talena-inc.com wrote: Hi Arun, Another feature that would be relevant and got deferred was the symlink work (HADOOP-10020) that Colin and Andrew were working on. Can we include this in hadoop-2.3.0 also? thanks hari On Sun, Nov 10, 2013 at 2:07 PM, Alejandro Abdelnur t...@cloudera.comwrote: Arun, thanks for jumping on this. On hadoop branch-2.2. I've quickly scanned the commit logs starting from the 2.2.0 release and I've found around 20 JIRAs that I like seeing in 2.2.1. Not all of them are bugs but the don't shake anything and improve usability. I presume others will have their own laundry lists as well and I wonder the union of all of them how much adds up to the current 81 commits. How about splitting the JIRAs among a few contributors to assert there is nothing risky in there? And if so get discuss getting rid of those commits for 2.2.1. IMO doing that would be cheaper than selectively applying commits on a fresh branch. Said this, I think we should get 2.2.1 out of the door before switching main efforts to 2.3.0. I volunteer myself to drive 2.2.1 a release if ASAP if you don't have the bandwidth at the moment for it. Cheers. Alejandro Commits in branch-2.2 that I'd like them to be in the 2.2.1 release: The ones prefixed with '*' technically are not bugs. YARN-1284. LCE: Race condition leaves dangling cgroups entries for killed containers. (Alejandro Abdelnur via Sandy Ryza) YARN-1265. Fair Scheduler chokes on unhealthy node reconnect (Sandy Ryza) YARN-1044. used/min/max resources do not display info in the scheduler page (Sangjin Lee via Sandy Ryza) YARN-305. Fair scheduler logs too many Node offered to app messages. (Lohit Vijayarenu via Sandy Ryza) *MAPREDUCE-5463. Deprecate SLOTS_MILLIS counters. (Tzuyoshi Ozawa via Sandy Ryza) YARN-1259. In Fair Scheduler web UI, queue num pending and num active apps switched. (Robert Kanter via Sandy Ryza) YARN-1295. In UnixLocalWrapperScriptBuilder, using bash -c can cause Text file busy errors. (Sandy Ryza) *MAPREDUCE-5457. Add a KeyOnlyTextOutputReader to enable streaming to write out text files without separators (Sandy Ryza) *YARN-1258. Allow configuring the Fair Scheduler root queue (Sandy Ryza) *YARN-1288. Make Fair Scheduler ACLs more user friendly (Sandy Ryza) YARN-1330. Fair Scheduler: defaultQueueSchedulingPolicy does not take effect (Sandy Ryza) HDFS-5403. WebHdfs client cannot communicate with older WebHdfs servers post HDFS-5306. Contributed by Aaron T. Myers. *YARN-1335. Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication (Sandy Ryza) *YARN-1333. Support blacklisting in the Fair Scheduler (Tsuyoshi Ozawa via Sandy Ryza) *MAPREDUCE-4680. Job history cleaner should only check timestamps of files in old enough directories (Robert Kanter via Sandy Ryza) YARN-1109. Demote NodeManager Sending out status for container logs to debug (haosdent via Sandy Ryza) *YARN-1321. Changed NMTokenCache to support both singleton and an instance usage. Contributed by Alejandro Abdelnur YARN-1343. NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs. (tucu) YARN-1381. Same relaxLocality appears twice in exception message of AMRMClientImpl#checkLocalityRelaxationConflict() (Ted Yu via Sandy Ryza) HADOOP-9898. Set SO_KEEPALIVE on all our sockets. Contributed by Todd Lipcon. YARN-1388. Fair Scheduler page always displays blank fair share (Liyin Liang via Sandy Ryza) On Fri, Nov 8, 2013 at 10:35 PM, Chris Nauroth cnaur...@hortonworks.com wrote: Arun, what are your thoughts on test-only patches? I know I've been merging a lot of Windows test stabilization patches down to branch-2.2. These can't rightly be called blockers, but they do improve dev experience, and there is no risk to product code. Chris Nauroth Hortonworks http://hortonworks.com/ On Fri, Nov 8, 2013 at 1:30 AM, Steve Loughran ste...@hortonworks.com wrote: On 8 November 2013 02:42, Arun C Murthy a...@hortonworks.com wrote: Gang, Thinking through the next couple of releases here, appreciate f/b. # hadoop-2.2.1 I was looking through commit logs and there is a *lot* of content here (81 commits as on 11/7). Some are features/improvements and some are fixes - it's really hard to distinguish what is
[jira] [Created] (HDFS-5501) Fix pendingReceivedRequests tracking in BPServiceActor
Arpit Agarwal created HDFS-5501:
---
Summary: Fix pendingReceivedRequests tracking in BPServiceActor
Key: HDFS-5501
URL: https://issues.apache.org/jira/browse/HDFS-5501
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
{{BPServiceActor#pendingReceivedRequests}} is not tracked correctly for
multiple storages. Previously this counter was treated more like a flag, i.e.
we only cared whether it was zero or greater than zero. With multiple storages
we need to be more precise in tracking this value to correctly determine when
to generate incremental block reports.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HDFS-5491) Update stored edits for HDFS-2832
[ https://issues.apache.org/jira/browse/HDFS-5491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5491. - Resolution: Fixed Fix Version/s: Heterogeneous Storage (HDFS-2832) Thanks Nicholas. I committed this. Update stored edits for HDFS-2832 - Key: HDFS-5491 URL: https://issues.apache.org/jira/browse/HDFS-5491 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Arpit Agarwal Assignee: Arpit Agarwal Priority: Minor Fix For: Heterogeneous Storage (HDFS-2832) Attachments: editsStored, h5491.01.patch, h5491.03.patch Update the stored edits binary and xml files to fix the following tests that rely on it: # TestOfflineEditsViewer # TestOfflineImageViewer # TestSnapshot -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: HDFS read/write data throttling
Hi, lohit. There is a Class named ThrottledInputStreamhttp://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java in hadoop-distcp, you could check it out and find more details. In addition to this, I am working on this and try to achieve resources control(include CPU, Network, Disk IO) in JVM. But my implementation is depends on cgroup, which only could run in Linux. I would push my library(java-cgroup) to github in the next several months. If you are interested at it, give my any advices and help me improve it please. :-) On Tue, Nov 12, 2013 at 3:47 AM, lohit lohit.vijayar...@gmail.com wrote: Hi Adam, Thanks for the reply. The changes I was referring was in FileSystem.java layer which should not affect HDFS Replication/NameNode operations. To give better idea this would affect clients something like this Configuration conf = new Configuration(); conf.setInt(read.bandwitdh.mbpersec, 20); // 20MB/s FileSystem fs = FileSystem.get(conf); FSDataInputStream fis = fs.open(/path/to/file.xt); fis.read(); // -- This would be max of 20MB/s 2013/11/11 Adam Muise amu...@hortonworks.com See https://issues.apache.org/jira/browse/HDFS-3475 Please note that this has met with many unexpected impacts on workload. Be careful and be mindful of your Datanode memory and network capacity. On Mon, Nov 11, 2013 at 1:59 PM, lohit lohit.vijayar...@gmail.com wrote: Hello Devs, Wanted to reach out and see if anyone has thought about ability to throttle data transfer within HDFS. One option we have been thinking is to throttle on a per FileSystem basis, similar to Statistics in FileSystem. This would mean anyone with handle to HDFS/Hftp will be throttled globally within JVM. Right value to come up for this would be based on type of hardware we use and how many tasks/clients we allow. On the other hand doing something like this at FileSystem layer would mean many other tasks such as Job jar copy, DistributedCache copy and any hidden data movement would also be throttled. We wanted to know if anyone has had such requirement on their clusters in the past and what was the thinking around it. Appreciate your inputs/comments -- Have a Nice Day! Lohit -- * Adam Muise * Solutions Engineer -- Phone:416-417-4037 Email: amu...@hortonworks.com Website: http://www.hortonworks.com/ * Follow Us: * http://facebook.com/hortonworks/?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature http://twitter.com/hortonworks?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature http://www.linkedin.com/company/hortonworks?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature [image: photo] Latest From Our Blog: How to use R and other non-Java languages in MapReduce and Hive http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Have a Nice Day! Lohit -- Best Regards, Haosdent Huang
[jira] [Created] (HDFS-5502) Fix HTTPS support for HsftpFileSystem
Haohui Mai created HDFS-5502: Summary: Fix HTTPS support for HsftpFileSystem Key: HDFS-5502 URL: https://issues.apache.org/jira/browse/HDFS-5502 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai The current implementation of HsftpFileSystem suffers from the following issues: * It initializes the SSLContext incorrectly. It blindly trusts all server certificates which creates a security hole. * It tries to cancel delegation token through http, not https, which leads to HDFS-5295. * It overrides the default socket factory for HttpsConnection. Given the fact that it trusts all server-side certificate, it accidentally disables all checks on server certificates for all https connections. This jira tracks the effort to fix the above issues. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-5494) Fix findbugs warnings for HDFS-2832
[ https://issues.apache.org/jira/browse/HDFS-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5494. - Resolution: Fixed Fix Version/s: Heterogeneous Storage (HDFS-2832) Hadoop Flags: Reviewed Thanks Nicholas! I committed this to branch HDFS-2832. Fix findbugs warnings for HDFS-2832 --- Key: HDFS-5494 URL: https://issues.apache.org/jira/browse/HDFS-5494 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: Heterogeneous Storage (HDFS-2832) Attachments: h5494.01.patch, h5494.02.patch, h5494.03.patch, h5494.04.patch Jenkins flagged the following findbugs warnings for the HDFS-2832 merge patch. https://builds.apache.org/job/PreCommit-HDFS-Build/5384//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: HDFS read/write data throttling
Hey Lohit, This is an interesting topic, and something I actually worked on in grad school before coming to Cloudera. It'd help if you could outline some of your usecases and how per-FileSystem throttling would help. For what I was doing, it made more sense to throttle on the DN side since you have a better view over all the I/O happening on the system, and you have knowledge of different volumes so you can set limits per-disk. This still isn't 100% reliable though since normally a portion of each disk is used for MR scratch space, which the DN doesn't have control over. I tried playing with thread I/O priorities here, but didn't see much improvement. Maybe the newer cgroups stuff can help out. I'm sure per-FileSystem throttling will have some benefits (and probably be easier than some DN-side implementation) but again, it'd help to better understand the problem you are trying to solve. Best, Andrew On Mon, Nov 11, 2013 at 6:16 PM, Haosong Huang haosd...@gmail.com wrote: Hi, lohit. There is a Class named ThrottledInputStream http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java in hadoop-distcp, you could check it out and find more details. In addition to this, I am working on this and try to achieve resources control(include CPU, Network, Disk IO) in JVM. But my implementation is depends on cgroup, which only could run in Linux. I would push my library(java-cgroup) to github in the next several months. If you are interested at it, give my any advices and help me improve it please. :-) On Tue, Nov 12, 2013 at 3:47 AM, lohit lohit.vijayar...@gmail.com wrote: Hi Adam, Thanks for the reply. The changes I was referring was in FileSystem.java layer which should not affect HDFS Replication/NameNode operations. To give better idea this would affect clients something like this Configuration conf = new Configuration(); conf.setInt(read.bandwitdh.mbpersec, 20); // 20MB/s FileSystem fs = FileSystem.get(conf); FSDataInputStream fis = fs.open(/path/to/file.xt); fis.read(); // -- This would be max of 20MB/s 2013/11/11 Adam Muise amu...@hortonworks.com See https://issues.apache.org/jira/browse/HDFS-3475 Please note that this has met with many unexpected impacts on workload. Be careful and be mindful of your Datanode memory and network capacity. On Mon, Nov 11, 2013 at 1:59 PM, lohit lohit.vijayar...@gmail.com wrote: Hello Devs, Wanted to reach out and see if anyone has thought about ability to throttle data transfer within HDFS. One option we have been thinking is to throttle on a per FileSystem basis, similar to Statistics in FileSystem. This would mean anyone with handle to HDFS/Hftp will be throttled globally within JVM. Right value to come up for this would be based on type of hardware we use and how many tasks/clients we allow. On the other hand doing something like this at FileSystem layer would mean many other tasks such as Job jar copy, DistributedCache copy and any hidden data movement would also be throttled. We wanted to know if anyone has had such requirement on their clusters in the past and what was the thinking around it. Appreciate your inputs/comments -- Have a Nice Day! Lohit -- * Adam Muise * Solutions Engineer -- Phone:416-417-4037 Email: amu...@hortonworks.com Website: http://www.hortonworks.com/ * Follow Us: * http://facebook.com/hortonworks/?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature http://twitter.com/hortonworks?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature http://www.linkedin.com/company/hortonworks?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature [image: photo] Latest From Our Blog: How to use R and other non-Java languages in MapReduce and Hive http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/?utm_source=WiseStamputm_medium=emailutm_term=utm_content=utm_campaign=signature -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --
Re: HDFS read/write data throttling
2013/11/11 Andrew Wang andrew.w...@cloudera.com Hey Lohit, This is an interesting topic, and something I actually worked on in grad school before coming to Cloudera. It'd help if you could outline some of your usecases and how per-FileSystem throttling would help. For what I was doing, it made more sense to throttle on the DN side since you have a better view over all the I/O happening on the system, and you have knowledge of different volumes so you can set limits per-disk. This still isn't 100% reliable though since normally a portion of each disk is used for MR scratch space, which the DN doesn't have control over. I tried playing with thread I/O priorities here, but didn't see much improvement. Maybe the newer cgroups stuff can help out. Thanks. Yes, we also thought about having something on DataNode. This would also mean one could easily throttle client who access from outside the cluster, for example distcp or hftp copies. Clients need not worry about throttle configs and each cluster can control how much much throughput can be achieved. We do want to have something like this. I'm sure per-FileSystem throttling will have some benefits (and probably be easier than some DN-side implementation) but again, it'd help to better understand the problem you are trying to solve. One idea was flexibility for client to override and have value they can set. For on trusted cluster we could allow clients to go beyond default value for some usecases. Alternatively we also thought about having default value and max value where clients could change default, but not go beyond default. Another problem with DN side config is having different values for different clients and easily changing those for selective clients. As, Haosong also suggested we could wrap FSDataOutputStream/FSDataInput stream with ThrottleInputStream. But we might have to be careful of any code which uses FileSystem APIs and accidentally throttling itself. (like reducer copy, distributed cache and such...) Best, Andrew On Mon, Nov 11, 2013 at 6:16 PM, Haosong Huang haosd...@gmail.com wrote: Hi, lohit. There is a Class named ThrottledInputStream http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java in hadoop-distcp, you could check it out and find more details. In addition to this, I am working on this and try to achieve resources control(include CPU, Network, Disk IO) in JVM. But my implementation is depends on cgroup, which only could run in Linux. I would push my library(java-cgroup) to github in the next several months. If you are interested at it, give my any advices and help me improve it please. :-) On Tue, Nov 12, 2013 at 3:47 AM, lohit lohit.vijayar...@gmail.com wrote: Hi Adam, Thanks for the reply. The changes I was referring was in FileSystem.java layer which should not affect HDFS Replication/NameNode operations. To give better idea this would affect clients something like this Configuration conf = new Configuration(); conf.setInt(read.bandwitdh.mbpersec, 20); // 20MB/s FileSystem fs = FileSystem.get(conf); FSDataInputStream fis = fs.open(/path/to/file.xt); fis.read(); // -- This would be max of 20MB/s 2013/11/11 Adam Muise amu...@hortonworks.com See https://issues.apache.org/jira/browse/HDFS-3475 Please note that this has met with many unexpected impacts on workload. Be careful and be mindful of your Datanode memory and network capacity. On Mon, Nov 11, 2013 at 1:59 PM, lohit lohit.vijayar...@gmail.com wrote: Hello Devs, Wanted to reach out and see if anyone has thought about ability to throttle data transfer within HDFS. One option we have been thinking is to throttle on a per FileSystem basis, similar to Statistics in FileSystem. This would mean anyone with handle to HDFS/Hftp will be throttled globally within JVM. Right value to come up for this would be based on type of hardware we use and how many tasks/clients we allow. On the other hand doing something like this at FileSystem layer would mean many other tasks such as Job jar copy, DistributedCache copy and any hidden data movement would also be throttled. We wanted to know if anyone has had such requirement on their clusters in the past and what was the thinking around it. Appreciate your inputs/comments -- Have a Nice Day! Lohit -- * Adam Muise * Solutions Engineer -- Phone:416-417-4037 Email: amu...@hortonworks.com Website: http://www.hortonworks.com/ * Follow Us: *
[jira] [Resolved] (HDFS-4642) Allow lease recovery for multiple paths to be issued in one request
[
https://issues.apache.org/jira/browse/HDFS-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HDFS-4642.
--
Resolution: Later
Allow lease recovery for multiple paths to be issued in one request
---
Key: HDFS-4642
URL: https://issues.apache.org/jira/browse/HDFS-4642
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ted Yu
Currently client can only request lease recovery for one Path:
{code}
public boolean recoverLease(Path f) throws IOException {
{code}
For HBase distributed log splitting, Nicolas made a suggestion here:
https://issues.apache.org/jira/browse/HBASE-7878?focusedCommentId=13615364page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13615364
HBase master collects the files that should be split, it issues lease
recovery for the files (in one request), then distribute log splitting.
This would help shorten MTTR.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
