On Fri, Apr 27, 2018 at 7:09 AM, Quanlong Huang <[email protected]> wrote:
> Thank you, Todd! But my IMPALA_HOME occupied 14GB after compilation... > > > The be directory contributes 4.4GB but I think only impalad and > libfesupport.so is needed. They're 981MB in total in a debug build. It's ok. > However, the toolchain directory contributes 8.5GB. It's hard to pick up > things we actually need (i.e. libkudu_client.so.0, libstdc++.so.6, > libgcc_s.so.1, and cdh_components, etc.). > I think I may not handle them in an elegant way. Could you give some more > advise? > > Maybe run 'ldd' on your impalad binary to figure out which shared objects are actually necessary? I also find that during development I have a good sense of which pieces actually changed. For example if I updated only the backend I only bother to re-copy impalad and not any of the jars. Vice versa, if I only updated something in the front end I'd only re-copy the FE jar to the cluster. That way you only pay the expensive deployment step the first time you set up your cluster. -Todd > At 2018-04-26 00:49:05, "Todd Lipcon" <[email protected]> wrote: > >Hi Quanlong, > > > >If you dont need full debuginfo on your cluster, you might consider > running > >'strip --strip-debug' on the impalad binary that you output. Between that > >and using 'rsync' instead of copying a new full directory of jars every > >time, it's usually not that many MB (<100?) I usually do builds on a > >machine in the same network as the machines I plan on deploying on so that > >the copy runs at several hundred MB/second, rather than building on my > >laptop far away. > > > >Tools like pscp and pssh are also handy of course. > > > >Hope that helps, > >-Todd > > > >On Wed, Apr 25, 2018 at 1:48 AM, Quanlong Huang <[email protected]> > >wrote: > > > >> Hi all, > >> > >> > >> Recently when I have a try on Impala-2.12.0-rc1, I find it really hard > to > >> deploy Impala manually (I used to do this by Cloudera Manager). The > >> directory size is huge after compiled so I only distributed something I > >> thought really needed. This work is tedious and prone to errors. > >> > >> > >> Is there a best practice for packaging and distributing the binaries > after > >> compiling? > >> > >> > >> Thanks, > >> Quanlong > > > > > > > > > >-- > >Todd Lipcon > >Software Engineer, Cloudera > -- Todd Lipcon Software Engineer, Cloudera
