Thanks for opening up the JIRA! It's pretty common for me to rebuild clean because I'm doing a lot development on both trunk and branch-2.8 and am constantly switching which one I'm compiling.
Eric On Wed, Mar 28, 2018 at 10:49 AM, Jim Clampffer <james.clampf...@gmail.com> wrote: > It's certainly possible to add a flag to disable the libhdfs++ build, I > opened HDFS-13362 for that. I can put up an initial patch that includes a > flag for cmake so that you could do -Dnative_cmake_args="skipLibhdfspp" > or something similar. I don't have a good understanding of maven and > antrun so if someone wants to help plumb a maven flag to the cmake flag I'd > appreciate it. > > Is it common to do completely clean builds for interactive use? > Incremental rebuilds of libhdfs++ shouldn't be adding more than a few > seconds to the process. > > On Tue, Mar 27, 2018 at 3:09 PM, Anu Engineer <aengin...@hortonworks.com> > wrote: > >> Would it be possible to add a maven flag like –skipShade, that helps in >> reducing the compile time for people who does not need to build libhdfs++ ? >> >> >> >> Thanks >> Anu >> >> >> >> >> >> *From: *Jim Clampffer <james.clampf...@gmail.com> >> *Date: *Tuesday, March 27, 2018 at 11:09 AM >> *To: *Eric Badger <ebad...@oath.com> >> *Cc: *Deepak Majeti <majeti.dee...@gmail.com>, Jitendra Pandey < >> jiten...@hortonworks.com>, Anu Engineer <aengin...@hortonworks.com>, >> Mukul Kumar Singh <msi...@hortonworks.com>, Owen O'Malley < >> owen.omal...@gmail.com>, Chris Douglas <cdoug...@apache.org>, Hdfs-dev < >> hdfs-dev@hadoop.apache.org> >> *Subject: *Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to >> trunk >> >> >> >> Hi Eric, >> >> There isn't a way to completely skip compiling libhdfs++ as part of the >> native build. You could pass >> -Dnative_cmake_args="-DHDFSPP_LIBRARY_ONLY=TRUE" >> to maven to avoid building all of the libhdfs++ tests, examples, and tools >> though. That cut the native client build time from 4:10 to 2:20 for me. >> >> -Jim >> >> >> >> On Tue, Mar 27, 2018 at 12:25 PM, Eric Badger <ebad...@oath.com> wrote: >> >> Is there a way to skip the libhdfs++ compilation during a native build? I >> just went to build native trunk to test out some container-executor changes >> and it spent 7:49 minutes out of 14:31 minutes in Apache Hadoop HDFS Native >> Client. For me it basically doubled the compilation time. >> >> >> >> Eric >> >> >> >> On Fri, Mar 16, 2018 at 4:01 PM, Deepak Majeti <majeti.dee...@gmail.com> >> wrote: >> >> Thanks for all your hard work on getting this feature (with > 200 >> sub-tasks) in James! >> >> On Fri, Mar 16, 2018 at 12:05 PM, Jim Clampffer < >> james.clampf...@gmail.com> >> wrote: >> >> >> > With 6 +1s, 0 0s, and 0 -1s the vote passes. I'll be merging this into >> > trunk shortly. >> > >> > Thanks everyone who participated in the discussion and vote! And many >> > thanks to everyone who contributed code and feedback throughout the >> > development process! Particularly Bob, Anatoli, Xiaowei and Deepak who >> > provided lots of large pieces of code as well as folks like Owen, Chris >> D, >> > Allen, and Stephen W who provided various support and guidance with the >> > Apache process and project design. >> > >> > On Wed, Mar 14, 2018 at 1:32 PM, Jitendra Pandey < >> jiten...@hortonworks.com >> > > >> > wrote: >> > >> > > +1 (binding) >> > > >> > > On 3/14/18, 9:57 AM, "Anu Engineer" <aengin...@hortonworks.com> >> wrote: >> > > >> > > +1 (binding). Thanks for all the hard work and getting this client >> > > ready. >> > > It is nice to have an official and supported native client for >> HDFS. >> > > >> > > Thanks >> > > Anu >> > > >> > > On 3/13/18, 8:16 PM, "Mukul Kumar Singh" <msi...@hortonworks.com> >> > > wrote: >> > > >> > > +1 (binding) >> > > >> > > Thanks, >> > > Mukul >> > > >> > > On 14/03/18, 2:06 AM, "Owen O'Malley" <owen.omal...@gmail.com >> > >> > > wrote: >> > > >> > > +1 (binding) >> > > >> > > .. Owen >> > > >> > > On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas < >> > > cdoug...@apache.org> wrote: >> > > >> > > > +1 (binding) -C >> > > > >> > > > On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer < >> > > james.clampf...@gmail.com> >> > > > wrote: >> > > > > Hi Everyone, >> > > > > >> > > > > The feedback was generally positive on the discussion >> > > thread [1] so I'd >> > > > > like to start a formal vote for merging HDFS-8707 >> > > (libhdfs++) into trunk. >> > > > > The vote will be open for 7 days and end 6PM EST on >> > > 3/15/18. >> > > > > >> > > > > This branch includes a C++ implementation of an HDFS >> > > client for use in >> > > > > applications that don't run an in-process JVM. Right >> now >> > > the branch only >> > > > > supports reads and metadata calls. >> > > > > >> > > > > Features (paraphrasing the list from the discussion >> > > thread): >> > > > > -Avoiding the JVM means applications that use >> libhdfs++ >> > > can explicitly >> > > > > control resources (memory, FDs, threads). The driving >> > > goal for this >> > > > > project was to let C/C++ applications access HDFS >> while >> > > maintaining a >> > > > > single heap. >> > > > > -Includes support for Kerberos authentication. >> > > > > -Includes a libhdfs/libhdfs3 compatible C API as well >> as >> > a >> > > C++ API that >> > > > > supports asynchronous operations. Applications that >> only >> > > do reads may be >> > > > > able to use this as a drop in replacement for libhdfs. >> > > > > -Asynchronous IO is built on top of boost::asio which >> in >> > > turn uses >> > > > > select/epoll so many sockets can be monitored from a >> > > single thread (or >> > > > > thread pool) rather than spawning a thread to sleep >> on a >> > > blocked socket. >> > > > > -Includes a set of utilities written in C++ that >> mirror >> > > the CLI tools >> > > > (e.g. >> > > > > ./hdfs dfs -ls). These have a 3 order of magnitude >> lower >> > > startup time >> > > > than >> > > > > java client which is useful for scripts that need to >> work >> > > with many >> > > > files. >> > > > > -Support for cancelable reads that release associated >> > > resources >> > > > > immediately. Useful for applications that need to be >> > > responsive to >> > > > > interactive users. >> > > > > >> > > > > Other points: >> > > > > -This is almost all new code in a new subdirectory. >> No >> > > Java source for >> > > > the >> > > > > rest of hadoop was changed so there's no risk of >> > > regressions there. The >> > > > > only changes outside of that subdirectory were >> > integrating >> > > the build in >> > > > > some of the pom files and adding a couple >> dependencies to >> > > the DockerFile. >> > > > > -The library has had plenty of burn-in time. It's >> been >> > > used in >> > > > production >> > > > > for well over a year and is indirectly being >> distributed >> > > as part of the >> > > > > Apache ORC project (in the form of a third party >> > > dependency). >> > > > > -There isn't much in the way of well formatted >> > > documentation right now. >> > > > > The documentation for the libhdfs API is applicable to >> > the >> > > libhdfs++ C >> > > > API. >> > > > > Header files describe various component including >> details >> > > about threading >> > > > > and lifecycle expectations for important objects. >> Good >> > > places to start >> > > > are >> > > > > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h >> > and >> > > rpc_enginel.h. >> > > > > >> > > > > I'll start with my +1 (binding). >> > > > > >> > > > > [1] >> > > > > http://mail-archives.apache. >> > org/mod_mbox/hadoop-hdfs-dev/ >> > > > 201803.mbox/browser >> > > > > (second message in thread, can't figure out how to >> link >> > > directly to mine) >> > > > > >> > > > > Thanks! >> > > > >> > > > ------------------------------ >> > ------------------------------ >> > > --------- >> > > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop. >> > > apache.org >> > > > For additional commands, e-mail: >> > > hdfs-dev-h...@hadoop.apache.org >> > > > >> > > > >> > > >> > > >> > > >> > > ------------------------------------------------------------ >> > > --------- >> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.ap >> ache.org >> > > For additional commands, e-mail: >> hdfs-dev-h...@hadoop.apache.org >> > > >> > > >> > > >> > > >> > > ------------------------------------------------------------ >> > --------- >> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > > >> > > >> > > >> > > >> > >> >> >> >> -- >> >> regards, >> Deepak Majeti >> >> >> >> >> > >
