date:20180313

Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk

2018-03-13 Thread Mukul Kumar Singh

+1 (binding)

Thanks,
Mukul

On 14/03/18, 2:06 AM, "Owen O'Malley"  wrote:

+1 (binding)

.. Owen

On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas  wrote:

> +1 (binding) -C
>
> On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer 
> wrote:
> > Hi Everyone,
> >
> > The feedback was generally positive on the discussion thread [1] so I'd
> > like to start a formal vote for merging HDFS-8707 (libhdfs++) into 
trunk.
> > The vote will be open for 7 days and end 6PM EST on 3/15/18.
> >
> > This branch includes a C++ implementation of an HDFS client for use in
> > applications that don't run an in-process JVM.  Right now the branch 
only
> > supports reads and metadata calls.
> >
> > Features (paraphrasing the list from the discussion thread):
> > -Avoiding the JVM means applications that use libhdfs++ can explicitly
> > control resources (memory, FDs, threads).  The driving goal for this
> > project was to let C/C++ applications access HDFS while maintaining a
> > single heap.
> > -Includes support for Kerberos authentication.
> > -Includes a libhdfs/libhdfs3 compatible C API as well as a C++ API that
> > supports asynchronous operations.  Applications that only do reads may 
be
> > able to use this as a drop in replacement for libhdfs.
> > -Asynchronous IO is built on top of boost::asio which in turn uses
> > select/epoll so many sockets can be monitored from a single thread (or
> > thread pool) rather than spawning a thread to sleep on a blocked socket.
> > -Includes a set of utilities written in C++ that mirror the CLI tools
> (e.g.
> > ./hdfs dfs -ls).  These have a 3 order of magnitude lower startup time
> than
> > java client which is useful for scripts that need to work with many
> files.
> > -Support for cancelable reads that release associated resources
> > immediately.  Useful for applications that need to be responsive to
> > interactive users.
> >
> > Other points:
> > -This is almost all new code in a new subdirectory.  No Java source for
> the
> > rest of hadoop was changed so there's no risk of regressions there.  The
> > only changes outside of that subdirectory were integrating the build in
> > some of the pom files and adding a couple dependencies to the 
DockerFile.
> > -The library has had plenty of burn-in time.  It's been used in
> production
> > for well over a year and is indirectly being distributed as part of the
> > Apache ORC project (in the form of a third party dependency).
> > -There isn't much in the way of well formatted documentation right now.
> > The documentation for the libhdfs API is applicable to the libhdfs++ C
> API.
> > Header files describe various component including details about 
threading
> > and lifecycle expectations for important objects.  Good places to start
> are
> > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h and 
rpc_enginel.h.
> >
> > I'll start with my +1 (binding).
> >
> > [1]
> > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/
> 201803.mbox/browser
> > (second message in thread, can't figure out how to link directly to 
mine)
> >
> > Thanks!
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDFS-13278) Correct the logic of mount validate to avoid the bad mountPoint

2018-03-13 Thread maobaolong (JIRA)

maobaolong created HDFS-13278:
-

 Summary: Correct the logic of mount validate to avoid the bad 
mountPoint
 Key: HDFS-13278
 URL: https://issues.apache.org/jira/browse/HDFS-13278
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs
Affects Versions: 3.2.0
Reporter: maobaolong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size

2018-03-13 Thread Bharat Viswanadham (JIRA)

Bharat Viswanadham created HDFS-13277:
-

 Summary: Improve move to account for usage (number of files) to 
limit trash dir size
 Key: HDFS-13277
 URL: https://issues.apache.org/jira/browse/HDFS-13277
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


The trash subdirectory maximum entries. This puts an upper limit on the size of 
subdirectories in replica-trash. Set this default value to blockinvalidateLimit.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13276) TestDistributedFileSystem doesn't cleanup MiniDFSCluster if test times out

2018-03-13 Thread JIRA

Íñigo Goiri created HDFS-13276:
--

 Summary: TestDistributedFileSystem doesn't cleanup MiniDFSCluster 
if test times out
 Key: HDFS-13276
 URL: https://issues.apache.org/jira/browse/HDFS-13276
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri
Assignee: Lukas Majercak


If a unit tests times out, it may leave a MiniDFSCluster behing. This is 
particularly bad in Windows where the new MiniDFSCluster cannot start and all 
tests will fail after this one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13275) Adding logs for BlockPoolManager#refreshNamenodes

2018-03-13 Thread Xiaoyu Yao (JIRA)

Xiaoyu Yao created HDFS-13275:
-

 Summary: Adding logs for BlockPoolManager#refreshNamenodes
 Key: HDFS-13275
 URL: https://issues.apache.org/jira/browse/HDFS-13275
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: Ajay Kumar


This helps detect and troubleshooting misconfigured NN service RPC or NN 
lifeline RPC address.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk

2018-03-13 Thread Owen O'Malley

+1 (binding)

.. Owen

On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas  wrote:

> +1 (binding) -C
>
> On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer 
> wrote:
> > Hi Everyone,
> >
> > The feedback was generally positive on the discussion thread [1] so I'd
> > like to start a formal vote for merging HDFS-8707 (libhdfs++) into trunk.
> > The vote will be open for 7 days and end 6PM EST on 3/15/18.
> >
> > This branch includes a C++ implementation of an HDFS client for use in
> > applications that don't run an in-process JVM.  Right now the branch only
> > supports reads and metadata calls.
> >
> > Features (paraphrasing the list from the discussion thread):
> > -Avoiding the JVM means applications that use libhdfs++ can explicitly
> > control resources (memory, FDs, threads).  The driving goal for this
> > project was to let C/C++ applications access HDFS while maintaining a
> > single heap.
> > -Includes support for Kerberos authentication.
> > -Includes a libhdfs/libhdfs3 compatible C API as well as a C++ API that
> > supports asynchronous operations.  Applications that only do reads may be
> > able to use this as a drop in replacement for libhdfs.
> > -Asynchronous IO is built on top of boost::asio which in turn uses
> > select/epoll so many sockets can be monitored from a single thread (or
> > thread pool) rather than spawning a thread to sleep on a blocked socket.
> > -Includes a set of utilities written in C++ that mirror the CLI tools
> (e.g.
> > ./hdfs dfs -ls).  These have a 3 order of magnitude lower startup time
> than
> > java client which is useful for scripts that need to work with many
> files.
> > -Support for cancelable reads that release associated resources
> > immediately.  Useful for applications that need to be responsive to
> > interactive users.
> >
> > Other points:
> > -This is almost all new code in a new subdirectory.  No Java source for
> the
> > rest of hadoop was changed so there's no risk of regressions there.  The
> > only changes outside of that subdirectory were integrating the build in
> > some of the pom files and adding a couple dependencies to the DockerFile.
> > -The library has had plenty of burn-in time.  It's been used in
> production
> > for well over a year and is indirectly being distributed as part of the
> > Apache ORC project (in the form of a third party dependency).
> > -There isn't much in the way of well formatted documentation right now.
> > The documentation for the libhdfs API is applicable to the libhdfs++ C
> API.
> > Header files describe various component including details about threading
> > and lifecycle expectations for important objects.  Good places to start
> are
> > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h and rpc_enginel.h.
> >
> > I'll start with my +1 (binding).
> >
> > [1]
> > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/
> 201803.mbox/browser
> > (second message in thread, can't figure out how to link directly to mine)
> >
> > Thanks!
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets

2018-03-13 Thread JIRA

Íñigo Goiri created HDFS-13274:
--

 Summary: RBF: Extend RouterRpcClient to use multiple sockets
 Key: HDFS-13274
 URL: https://issues.apache.org/jira/browse/HDFS-13274
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Íñigo Goiri






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13273) Fix compilation issue in trunk

2018-03-13 Thread Bharat Viswanadham (JIRA)

Bharat Viswanadham created HDFS-13273:
-

 Summary: Fix compilation issue in trunk
 Key: HDFS-13273
 URL: https://issues.apache.org/jira/browse/HDFS-13273
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-trunk-Commit/source/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatusWithECPolicy.java:[40,8]
 class TestFileStatusWithDefaultECPolicy is public, should be declared in a 
file named TestFileStatusWithDefaultECPolicy.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13272) DataNodeHttpServer hard-codes HttpServer2 threads at 10

2018-03-13 Thread Erik Krogen (JIRA)

Erik Krogen created HDFS-13272:
--

 Summary: DataNodeHttpServer hard-codes HttpServer2 threads at 10
 Key: HDFS-13272
 URL: https://issues.apache.org/jira/browse/HDFS-13272
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Reporter: Erik Krogen
Assignee: Erik Krogen


In HDFS-7279, the Jetty server on the DataNode was hard-coded to use 10 
threads. In addition to the possibility of this being too few threads, it is 
much higher than necessary in resource constrained environments such as 
MiniDFSCluster. To avoid compatibility issues, rather than using 
{{HttpServer2#HTTP_MAX_THREADS}} directly, we can introduce a new configuration 
for the DataNode's thread pool size.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-03-13 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/

[Mar 12, 2018 2:01:42 PM] (stevel) HADOOP-15297. Make S3A etag => checksum 
feature optional. Contributed by
[Mar 12, 2018 4:58:56 PM] (inigoiri) HDFS-11399. Many tests fails in Windows 
due to injecting disk failures.
[Mar 12, 2018 7:07:22 PM] (aengineer) HDFS-10618. 
TestPendingReconstruction#testPendingAndInvalidate is flaky
[Mar 12, 2018 8:43:27 PM] (cdouglas) HADOOP-14742. Document multi-URI 
replication Inode for ViewFS.
[Mar 12, 2018 9:29:44 PM] (cdouglas) HDFS-12677. Extend 
TestReconstructStripedFile with a random EC policy.
[Mar 12, 2018 11:35:26 PM] (yufei) YARN-8024. LOG in class 
MaxRunningAppsEnforcer is initialized with a
[Mar 13, 2018 12:28:15 AM] (inigoiri) HDFS-13241. RBF: TestRouterSafemode 
failed if the port  is in use.




-1 overall


The following subsystems voted -1:
findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
 
   Nullcheck of callerUGI at line 146 of value previously dereferenced in 
org.apache.hadoop.yarn.server.webapp.AppBlock.render(HtmlBlock$Block) At 
AppBlock.java:146 of value previously dereferenced in 
org.apache.hadoop.yarn.server.webapp.AppBlock.render(HtmlBlock$Block) At 
AppBlock.java:[line 146] 

Failed junit tests :

   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.server.federation.router.TestRouterSafemode 
   hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.yarn.sls.TestSLSStreamAMSynth 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-compile-javac-root.txt
  [292K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/whitespace-eol.txt
  [9.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/whitespace-tabs.txt
  [288K]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [328K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [48K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [12K]

Powered by Apache Yetus 0.8.0-SNAPSHOT

[jira] [Created] (HDFS-13271) WebHDFS: Add constructor in SnapshottableDirectoryStatus with HdfsFileStatus as argument

2018-03-13 Thread Lokesh Jain (JIRA)

Lokesh Jain created HDFS-13271:
--

 Summary: WebHDFS: Add constructor in SnapshottableDirectoryStatus 
with HdfsFileStatus as argument
 Key: HDFS-13271
 URL: https://issues.apache.org/jira/browse/HDFS-13271
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Reporter: Lokesh Jain
Assignee: Lokesh Jain


This jira aims to add a constructor in SnapshottableDirectoryStatus which takes 
HdfsFileStatus as a argument. This constructor will be used in 
JsonUtilClient#toSnapshottableDirectoryStatus for creating a 
SnapshottableDirectoryStatus object.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13270) RBF: Router audit logger

2018-03-13 Thread maobaolong (JIRA)

maobaolong created HDFS-13270:
-

 Summary: RBF: Router audit logger
 Key: HDFS-13270
 URL: https://issues.apache.org/jira/browse/HDFS-13270
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: hdfs
Affects Versions: 3.2.0
Reporter: maobaolong


We can use router auditlogger to log the client info and cmd, because the 
FSNamesystem#Auditlogger's log think the client are all from router.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk

[jira] [Created] (HDFS-13278) Correct the logic of mount validate to avoid the bad mountPoint

[jira] [Created] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size

[jira] [Created] (HDFS-13276) TestDistributedFileSystem doesn't cleanup MiniDFSCluster if test times out

[jira] [Created] (HDFS-13275) Adding logs for BlockPoolManager#refreshNamenodes

Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk

[jira] [Created] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets

[jira] [Created] (HDFS-13273) Fix compilation issue in trunk

[jira] [Created] (HDFS-13272) DataNodeHttpServer hard-codes HttpServer2 threads at 10

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[jira] [Created] (HDFS-13271) WebHDFS: Add constructor in SnapshottableDirectoryStatus with HdfsFileStatus as argument

[jira] [Created] (HDFS-13270) RBF: Router audit logger

12 matches

Site Navigation

Mail list logo

Footer information