[jira] [Resolved] (HDFS-13288) Why we don't add a harder lease expiration limit.

[Igloo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igloo resolved HDFS-13288.
--
Resolution: Invalid

> Why we don't add a harder lease expiration limit.
> -
>
> Key: HDFS-13288
> URL: https://issues.apache.org/jira/browse/HDFS-13288
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.6.5
>Reporter: Igloo
>Priority: Minor
>
> Currently there exists a soft expire timeout(1 minutes by default) and hard 
> expire timeout(60 minutes by default). 
> On our production environment. Some client began writing a file long 
> time(more than one year) ago, when writing finished and tried to close the 
> output stream, the client failed closing it (for some IOException. etc. ).  
> But the client process is a background service, it doesn't exit. So the lease 
> doesn't released for more than one year.
> The problem is that, the lease for the file is occupied, we have to call 
> recover lease on the file when doing demission or appending operation.
>  
> So I am wondering why we don't add a more harder lease expire timeout, when a 
> lease lasts too long (maybe one month),  revoke it. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13289) TestConnectionManager test case need correction

[Dibyendu Karmakar (JIRA)]

Dibyendu Karmakar created HDFS-13289:


 Summary: TestConnectionManager test case need correction
 Key: HDFS-13289
 URL: https://issues.apache.org/jira/browse/HDFS-13289
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Dibyendu Karmakar


In TestConnectionManager.testCleanup() 

 
{code:java}
// Make sure the number of connections doesn't go below minSize
ConnectionPool pool3 = new ConnectionPool(
conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
addConnectionsToPool(pool3, 10, 0);
poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
connManager.cleanup(pool3);
checkPoolConnections(TEST_USER3, 2, 0);
{code}
this part need correction.

 

Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections 
is done using TEST_USER3. 

In checkPoolConnections method 
{code:java}
if (e.getKey().getUgi() == ugi)
{code}
then only it will validate numOfConns and numOfActiveConns. In this case for 
TEST_USER3  ' *if*  'condition is returning *false* and if you pass any value 
to the checkPoolConnections method, the test case will pass.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13288) Why we don't add a harder lease expiration limit.

[Igloo (JIRA)]

Igloo created HDFS-13288:


 Summary: Why we don't add a harder lease expiration limit.
 Key: HDFS-13288
 URL: https://issues.apache.org/jira/browse/HDFS-13288
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 2.6.5
Reporter: Igloo


Currently there exists a soft expire timeout(1 minutes by default) and hard 
expire timeout(60 minutes by default). 

On our production environment. Some client began writing a file long time(more 
than one year) ago, when writing finished and tried to close the output stream, 
the client failed closing it (for some IOException. etc. ).  But the client 
process is a background service, it doesn't exit. So the lease doesn't released 
for more than one year.

The problem is the lease for the file is occupied, we have to call recover 
lease on the file.

So I am wondering why we don't add a more harder lease expire timeout, when a 
lease lasts too long (maybe one month), revoke it.  

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone

[Virajith Jalaparti (JIRA)]

Virajith Jalaparti created HDFS-13287:
-

 Summary: TestINodeFile#testGetBlockType results in NPE when run 
alone
 Key: HDFS-13287
 URL: https://issues.apache.org/jira/browse/HDFS-13287
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Virajith Jalaparti
Assignee: Virajith Jalaparti


When TestINodeFile#testGetBlockType is run by itself, it results in the 
following error:

{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218 s 
<<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestINodeFile
[ERROR] testGetBlockType(org.apache.hadoop.hdfs.server.namenode.TestINodeFile)  
Time elapsed: 0.023 s  <<< ERROR!
java.lang.NullPointerException
at 
org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getPolicyInfoByID(ErasureCodingPolicyManager.java:220)
at 
org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getByID(ErasureCodingPolicyManager.java:208)
at 
org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(INodeFile.java:207)
at 
org.apache.hadoop.hdfs.server.namenode.INodeFile.(INodeFile.java:266)
at 
org.apache.hadoop.hdfs.server.namenode.TestINodeFile.createStripedINodeFile(TestINodeFile.java:112)
at 
org.apache.hadoop.hdfs.server.namenode.TestINodeFile.testGetBlockType(TestINodeFile.java:299)

{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13286) Add haadmin commands to transition between standby and observer

[Chao Sun (JIRA)]

Chao Sun created HDFS-13286:
---

 Summary: Add haadmin commands to transition between standby and 
observer
 Key: HDFS-13286
 URL: https://issues.apache.org/jira/browse/HDFS-13286
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chao Sun
Assignee: Chao Sun


As discussed in HDFS-12975, we should allow explicit transition between standby 
and observer through haadmin command, such as:
{code}
haadmin -transitionToObserver
{code}

Initially we should support transition from observer to standby, and standby to 
observer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13285) Improve runtime for TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks

[Ajay Kumar (JIRA)]

Ajay Kumar created HDFS-13285:
-

 Summary: Improve runtime for 
TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks 
 Key: HDFS-13285
 URL: https://issues.apache.org/jira/browse/HDFS-13285
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ajay Kumar


TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks takes 
anywhere b/w 2-4 minutes depending on host machine. Jira intends to make it 
leaner.

cc: [~elgoiri]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/378/

[Oct 12, 2017 5:38:58 PM] (carlo curino) YARN-7317. Fix overallocation resulted 
from ceiling in
[Oct 12, 2017 5:59:20 PM] (ctrezzo) MAPREDUCE-5951. Add support for the YARN 
Shared Cache.
[Oct 13, 2017 9:00:25 AM] (aajisaka) Revert "HADOOP-13514. Upgrade maven 
surefire plugin to 2.19.1."
[Oct 13, 2017 9:47:05 AM] (rohithsharmaks) YARN-7180. Remove class 
ResourceType. Contributed by Sunil G.
[Oct 13, 2017 7:41:59 PM] (yufei) YARN-7270. Fix unsafe casting from long to 
int for class Resource and
[Oct 13, 2017 8:52:58 PM] (rkanter) HADOOP-14938. 
Configuration.updatingResource map should be initialized
[Oct 13, 2017 9:22:21 PM] (arp) HDFS-12553. Add nameServiceId to 
QJournalProtocol. Contributed by Bharat
[Oct 13, 2017 10:49:21 PM] (haibochen) YARN-7310. 
TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler.
[Oct 14, 2017 12:43:21 AM] (manojpec) HADOOP-13055. Implement linkMergeSlash 
and linkFallback for
[Oct 14, 2017 5:41:58 PM] (wangda) Addendum fix for: YARN-7269. Tracking URL in 
the app state does not get
[Oct 16, 2017 2:28:22 AM] (xiao) HDFS-12659. Update 
TestDeadDatanode#testNonDFSUsedONDeadNodeReReg to
[Oct 16, 2017 3:01:47 PM] (stevel) HADOOP-14935. Azure: POSIX permissions are 
taking effect in access()
[Oct 16, 2017 4:51:10 PM] (xiao) HDFS-12603. Enable async edit logging by 
default. Contributed by Andrew
[Oct 16, 2017 4:55:22 PM] (xiao) HDFS-12637. Extend 
TestDistributedFileSystemWithECFile with a random EC
[Oct 16, 2017 5:34:06 PM] (xiao) HDFS-12642. Log block and datanode details in 
BlockRecoveryWorker.
[Oct 16, 2017 7:57:48 PM] (weichiu) HDFS-12485. expunge may fail to remove 
trash from encryption zone.
[Oct 16, 2017 8:12:15 PM] (arun suresh) YARN-7275. NM Statestore cleanup for 
Container updates. (Kartheek
[Oct 16, 2017 8:57:03 PM] (junping_du) YARN-7124. LogAggregationTFileController 
deletes/renames while file is
[Oct 16, 2017 9:19:31 PM] (xiao) HADOOP-14949. TestKMS#testACLs fails 
intermittently.
[Oct 16, 2017 10:00:38 PM] (nroberts) YARN-7333. container-executor fails to 
remove entries from a directory
[Oct 16, 2017 10:34:32 PM] (rkanter) YARN-7308. TestApplicationACLs fails with 
FairScheduler (rkanter)
[Oct 16, 2017 11:42:59 PM] (weichiu) HADOOP-14948. Document missing config key 
hadoop.treat.subject.external.
[Oct 17, 2017 12:42:41 AM] (manojpec) HDFS-12614. 
FSPermissionChecker#getINodeAttrs() throws NPE when
[Oct 17, 2017 2:44:30 AM] (lei) HDFS-12613. Native EC coder should implement 
release() as idempotent
[Oct 17, 2017 5:15:53 PM] (haibochen) YARN-7341. 
TestRouterWebServiceUtil#testMergeMetrics is flakey. (Robert
[Oct 17, 2017 7:38:06 PM] (subu) YARN-7311. Fix TestRMWebServicesReservation 
parametrization for fair
[Oct 17, 2017 10:52:09 PM] (lei) HDFS-12612. DFSStripedOutputStream.close will 
throw if called a second
[Oct 17, 2017 11:04:19 PM] (haibochen) YARN-6546. SLS is slow while loading 10k 
queues. (Yufei Gu via Haibo
[Oct 18, 2017 2:06:45 AM] (xiao) HADOOP-14944. Add JvmMetrics to KMS.
[Oct 18, 2017 2:18:39 AM] (aajisaka) MAPREDUCE-6972. Enable try-with-resources 
for RecordReader. Contributed
[Oct 18, 2017 10:06:30 PM] (junping_du) HADOOP-14958. Fix source-level 
compatibility after HADOOP-11252.
[Oct 19, 2017 6:51:24 AM] (zhz) HDFS-12502. nntop should support a category 
based on
[Oct 19, 2017 1:02:13 PM] (weichiu) HADOOP-14880. [KMS] Document missing 
KMS client side configs.
[Oct 19, 2017 1:17:59 PM] (weichiu) HDFS-12619. Do not catch and throw 
unchecked exceptions if IBRs fail to
[Oct 19, 2017 8:25:08 PM] (haibochen) HADOOP-14771. hadoop-client does not 
include hadoop-yarn-client. (Ajay
[Oct 19, 2017 9:44:42 PM] (wangda) YARN-7338. Support same origin policy for 
cross site scripting
[Oct 19, 2017 9:45:44 PM] (wangda) YARN-7345. GPU Isolation: Incorrect minor 
device numbers written to
[Oct 19, 2017 11:39:25 PM] (yufei) YARN-7294. 
TestSignalContainer#testSignalRequestDeliveryToNM fails
[Oct 19, 2017 11:45:18 PM] (cdouglas) HADOOP-14816. Update Dockerfile to use 
Xenial. Contributed by Allen
[Oct 19, 2017 11:51:47 PM] (yufei) YARN-7359. 
TestAppManager.testQueueSubmitWithNoPermission() should be
[Oct 20, 2017 1:08:45 AM] (inigoiri) HDFS-12620. Backporting HDFS-10467 to 
branch-2. Contributed by Inigo
[Oct 20, 2017 1:42:04 AM] (kai.zheng) HDFS-12448. Make sure user defined 
erasure coding policy ID will not
[Oct 20, 2017 4:58:40 AM] (wangda) YARN-7170. Improve bower dependencies for 
YARN UI v2. (Sunil G via
[Oct 20, 2017 8:32:20 AM] (yufei) YARN-4090. Make Collections.sort() more 
efficient by caching resource
[Oct 20, 2017 4:02:06 PM] (eyang) YARN-7353. Improved volume mount check for 
directories and unit test
[Oct 20, 2017 5:00:13 PM] (yufei) YARN-7261. Add debug message for better 
download latency monitoring.
[Oct 20, 2017 6:15:20 PM] (yufei) YARN-7355. TestDistributedShell should be 
scheduler agnostic.
[Oct 20, 2017 8:27:21 PM] (wang) HDFS-12497. 

[jira] [Created] (HDFS-13284) Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY

[Lukas Majercak (JIRA)]

Lukas Majercak created HDFS-13284:
-

 Summary: Adjust criteria for 
LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY
 Key: HDFS-13284
 URL: https://issues.apache.org/jira/browse/HDFS-13284
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs, namenode
Reporter: Lukas Majercak
Assignee: Lukas Majercak






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: About reset branch-3.1 to trunk before release.

[Vinod Kumar Vavilapalli]

I see one new feature: https://issues.apache.org/jira/browse/YARN-7626: Allow 
regular expression matching in container-executor.cfg for devices and named 
docker volumes mount.

There are 21 sub-tasks. There are three feature-type JIRAs in those - 
https://issues.apache.org/jira/browse/YARN-7972, 
https://issues.apache.org/jira/browse/YARN-7891 and 
https://issues.apache.org/jira/browse/YARN-5015. These should be okay - not 
major disrupting features.

Everything else is either a bug-fix or an improvement so we should be good.

From the list, it doesn't look like resetting will destabilize 3.1, +1 for 
doing this.

Thanks
+Vinod

> On Mar 14, 2018, at 1:54 PM, Wangda Tan  wrote:
> 
> Hi mapreduce/yarn/common/hdfs-devs, 
> 
> As of now, we have all blockers done for 3.1.0 release [1]. The release is 
> running behind schedule due to a few security-related issues. Because of this 
> and since branch-3.1 is cut 5 weeks before on Feb 8, trunk 3.2 is already 
> diverging. There're 64 commits in trunk but not in branch-3.1. [2]
> 
> I took a quick scan of them, most of them are good fixes which we should 
> bring to 3.1.0 as well. And this can also reduce differences between 3.2.0 
> and 3.1.0 release for less maintenance burden in the future.
> 
> Unless anyone objects, we will reset branch-3.1 to trunk in 1-2 days and cut 
> RC after that.
> 
> Thoughts?
> 
> - Wangda
> 
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, 
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.0 ORDER BY 
> priority DESC
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) AND 
> fixVersion not in (3.1.0)


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode

[Lukas Majercak (JIRA)]

Lukas Majercak created HDFS-13283:
-

 Summary: Percentage based Reserved Space Calculation for DataNode
 Key: HDFS-13283
 URL: https://issues.apache.org/jira/browse/HDFS-13283
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: datanode, hdfs
Reporter: Lukas Majercak
Assignee: Lukas Majercak






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

About reset branch-3.1 to trunk before release.

[Wangda Tan]

Hi mapreduce/yarn/common/hdfs-devs,

As of now, we have all blockers done for 3.1.0 release [1]. The release is
running behind schedule due to a few security-related issues. Because of
this and since branch-3.1 is cut 5 weeks before on Feb 8, trunk 3.2 is
already diverging. There're 64 commits in trunk but not in branch-3.1. [2]

I took a quick scan of them, most of them are good fixes which we should
bring to 3.1.0 as well. And this can also reduce differences between 3.2.0
and 3.1.0 release for less maintenance burden in the future.

Unless anyone objects, we will reset branch-3.1 to trunk in 1-2 days and
cut RC after that.

Thoughts?

- Wangda

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.0 ORDER
BY priority DESC
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0)

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Owen O'Malley]

This discussion seems to have died down coming closer consensus without a
resolution.

I'd like to propose the following compromise:

* HDSL become a subproject of Hadoop.
* HDSL will release separately from Hadoop. Hadoop releases will not
contain HDSL and vice versa.
* HDSL will get its own jira instance so that the release tags stay
separate.
* On trunk (as opposed to release branches) HDSL will be a separate module
in Hadoop's source tree. This will enable the HDSL to work on their trunk
and the Hadoop trunk without making releases for every change.
* Hadoop's trunk will only build HDSL if a non-default profile is enabled.
* When Hadoop creates a release branch, the RM will delete the HDSL module
from the branch.
* HDSL will have their own Yetus checks and won't cause failures in the
Hadoop patch check.

I think this accomplishes most of the goals of encouraging HDSL development
while minimizing the potential for disruption of HDFS development.

Thoughts? Andrew, Jitendra, & Sanjay?

Thanks,
   Owen

[jira] [Created] (HDFS-13282) Create UT for ReplicaFileDeleteTask without minidfs cluster

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created HDFS-13282:
-

 Summary: Create UT for ReplicaFileDeleteTask without minidfs 
cluster
 Key: HDFS-13282
 URL: https://issues.apache.org/jira/browse/HDFS-13282
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


Address one of the review comment given in HDFS-13163
{quote}Let's try to replace the UT with an isolated unit test for 
ReplicaFileDeleteTask#run that doesn't spin up a MiniDfsCluster. Okay to do 
this in a separate Jira.
{quote}
We will not replace UT, as the other test is testing the behavior with actual 
delete happening.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk

[Jitendra Pandey]

+1 (binding)

On 3/14/18, 9:57 AM, "Anu Engineer"  wrote:

+1 (binding). Thanks for all the hard work and getting this client ready. 
It is nice to have an official and supported native client for HDFS.

Thanks
Anu

On 3/13/18, 8:16 PM, "Mukul Kumar Singh"  wrote:

+1 (binding)

Thanks,
Mukul

On 14/03/18, 2:06 AM, "Owen O'Malley"  wrote:

+1 (binding)

.. Owen

On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas 
 wrote:

> +1 (binding) -C
>
> On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer 

> wrote:
> > Hi Everyone,
> >
> > The feedback was generally positive on the discussion thread 
[1] so I'd
> > like to start a formal vote for merging HDFS-8707 (libhdfs++) 
into trunk.
> > The vote will be open for 7 days and end 6PM EST on 3/15/18.
> >
> > This branch includes a C++ implementation of an HDFS client for 
use in
> > applications that don't run an in-process JVM.  Right now the 
branch only
> > supports reads and metadata calls.
> >
> > Features (paraphrasing the list from the discussion thread):
> > -Avoiding the JVM means applications that use libhdfs++ can 
explicitly
> > control resources (memory, FDs, threads).  The driving goal for 
this
> > project was to let C/C++ applications access HDFS while 
maintaining a
> > single heap.
> > -Includes support for Kerberos authentication.
> > -Includes a libhdfs/libhdfs3 compatible C API as well as a C++ 
API that
> > supports asynchronous operations.  Applications that only do 
reads may be
> > able to use this as a drop in replacement for libhdfs.
> > -Asynchronous IO is built on top of boost::asio which in turn 
uses
> > select/epoll so many sockets can be monitored from a single 
thread (or
> > thread pool) rather than spawning a thread to sleep on a 
blocked socket.
> > -Includes a set of utilities written in C++ that mirror the CLI 
tools
> (e.g.
> > ./hdfs dfs -ls).  These have a 3 order of magnitude lower 
startup time
> than
> > java client which is useful for scripts that need to work with 
many
> files.
> > -Support for cancelable reads that release associated resources
> > immediately.  Useful for applications that need to be 
responsive to
> > interactive users.
> >
> > Other points:
> > -This is almost all new code in a new subdirectory.  No Java 
source for
> the
> > rest of hadoop was changed so there's no risk of regressions 
there.  The
> > only changes outside of that subdirectory were integrating the 
build in
> > some of the pom files and adding a couple dependencies to the 
DockerFile.
> > -The library has had plenty of burn-in time.  It's been used in
> production
> > for well over a year and is indirectly being distributed as 
part of the
> > Apache ORC project (in the form of a third party dependency).
> > -There isn't much in the way of well formatted documentation 
right now.
> > The documentation for the libhdfs API is applicable to the 
libhdfs++ C
> API.
> > Header files describe various component including details about 
threading
> > and lifecycle expectations for important objects.  Good places 
to start
> are
> > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h and 
rpc_enginel.h.
> >
> > I'll start with my +1 (binding).
> >
> > [1]
> > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/
> 201803.mbox/browser
> > (second message in thread, can't figure out how to link 
directly to mine)
> >
> > Thanks!
>
> 
-
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk

[Anu Engineer]

+1 (binding). Thanks for all the hard work and getting this client ready. 
It is nice to have an official and supported native client for HDFS.

Thanks
Anu

On 3/13/18, 8:16 PM, "Mukul Kumar Singh"  wrote:

+1 (binding)

Thanks,
Mukul

On 14/03/18, 2:06 AM, "Owen O'Malley"  wrote:

+1 (binding)

.. Owen

On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas  
wrote:

> +1 (binding) -C
>
> On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer 

> wrote:
> > Hi Everyone,
> >
> > The feedback was generally positive on the discussion thread [1] so 
I'd
> > like to start a formal vote for merging HDFS-8707 (libhdfs++) into 
trunk.
> > The vote will be open for 7 days and end 6PM EST on 3/15/18.
> >
> > This branch includes a C++ implementation of an HDFS client for use 
in
> > applications that don't run an in-process JVM.  Right now the 
branch only
> > supports reads and metadata calls.
> >
> > Features (paraphrasing the list from the discussion thread):
> > -Avoiding the JVM means applications that use libhdfs++ can 
explicitly
> > control resources (memory, FDs, threads).  The driving goal for this
> > project was to let C/C++ applications access HDFS while maintaining 
a
> > single heap.
> > -Includes support for Kerberos authentication.
> > -Includes a libhdfs/libhdfs3 compatible C API as well as a C++ API 
that
> > supports asynchronous operations.  Applications that only do reads 
may be
> > able to use this as a drop in replacement for libhdfs.
> > -Asynchronous IO is built on top of boost::asio which in turn uses
> > select/epoll so many sockets can be monitored from a single thread 
(or
> > thread pool) rather than spawning a thread to sleep on a blocked 
socket.
> > -Includes a set of utilities written in C++ that mirror the CLI 
tools
> (e.g.
> > ./hdfs dfs -ls).  These have a 3 order of magnitude lower startup 
time
> than
> > java client which is useful for scripts that need to work with many
> files.
> > -Support for cancelable reads that release associated resources
> > immediately.  Useful for applications that need to be responsive to
> > interactive users.
> >
> > Other points:
> > -This is almost all new code in a new subdirectory.  No Java source 
for
> the
> > rest of hadoop was changed so there's no risk of regressions there. 
 The
> > only changes outside of that subdirectory were integrating the 
build in
> > some of the pom files and adding a couple dependencies to the 
DockerFile.
> > -The library has had plenty of burn-in time.  It's been used in
> production
> > for well over a year and is indirectly being distributed as part of 
the
> > Apache ORC project (in the form of a third party dependency).
> > -There isn't much in the way of well formatted documentation right 
now.
> > The documentation for the libhdfs API is applicable to the 
libhdfs++ C
> API.
> > Header files describe various component including details about 
threading
> > and lifecycle expectations for important objects.  Good places to 
start
> are
> > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h and 
rpc_enginel.h.
> >
> > I'll start with my +1 (binding).
> >
> > [1]
> > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/
> 201803.mbox/browser
> > (second message in thread, can't figure out how to link directly to 
mine)
> >
> > Thanks!
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13281) Namenode#createFile should be /.reserved/raw/ aware.

[Rushabh S Shah (JIRA)]

Rushabh S Shah created HDFS-13281:
-

 Summary: Namenode#createFile should be /.reserved/raw/ aware.
 Key: HDFS-13281
 URL: https://issues.apache.org/jira/browse/HDFS-13281
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: encryption
Affects Versions: 2.8.3
Reporter: Rushabh S Shah


If I want to write to /.reserved/raw/ and if that directory happens to be 
in EZ, then namenode should not encrypt and copy the raw bytes from the source.
Namenode#startFileInt should be /.reserved/raw/ aware.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13280) WebHDFS: Fix NPE in get snasphottable directory list call

[Lokesh Jain (JIRA)]

Lokesh Jain created HDFS-13280:
--

 Summary: WebHDFS: Fix NPE in get snasphottable directory list call
 Key: HDFS-13280
 URL: https://issues.apache.org/jira/browse/HDFS-13280
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Reporter: Lokesh Jain
Assignee: Lokesh Jain


WebHdfs throws NPE when snapshottable directory status list is null.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13279) Datanodes usage is imbalanced if node

[Tao Jie (JIRA)]

Tao Jie created HDFS-13279:
--

 Summary: Datanodes usage is imbalanced if node
 Key: HDFS-13279
 URL: https://issues.apache.org/jira/browse/HDFS-13279
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Tao Jie






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org