OKAY. If snappy compression is often required by many users, then I think there's no reason holding us from not including it by default. If we're going this way, then another question is do we want to include the change in 1.0? If this does not introduce too much effort to release process, then I'm ok with it because those bigtop/slaves image somehow need to rebuild because of BIGTOP-1896 <https://issues.apache.org/jira/browse/BIGTOP-1896>, so I can include snappy as well.
Cos would you like to comment this in regards to the 1.0 release? 2015-06-11 13:21 GMT+08:00 김영우 <[email protected]>: > Snappy itself is one of the widely used compression codec for Hadoop and > it's ecosystem. Hadoop distros like CDH and HDP have snappy support by > default and provide related docs. > > IMO, If Hadoop in Bigtop has support snappy by default, we should include > it in bigtop_toolchain *explicitly* to avoid confusion. > > Thanks, > Youngwoo > > On Thu, Jun 11, 2015 at 5:28 AM, Olaf Flebbe <[email protected]> wrote: > >> Hi Evans, >> >> I see no essential need for snappy, too. If we do the CI build with >> snappy, than we rather should include it in bigtop_toolchain. >> >> Maybe it would be better to include snappy in an option puppet manifest >> rather to hardcode it in packages.pp. Feel free to reject my patch in the >> JIRA optionally with proposing a different solution. >> >> Olaf >> >> >> >> > Am 10.06.2015 um 03:10 schrieb Evans Ye <[email protected]>: >> > >> > Thanks for the comment Olaf. >> > Whether or not to include snappy codec is optional in upstream hadoop >> projrct, therefore I'm not quite sure to put it in to toolchain is the >> right way to go. Since this implies that we'll build hadoop with snappy by >> default. >> > Do you see most of your users or customer require this feature? >> > >> > 2015/6/10 上午4:13 於 "Olaf Flebbe" <[email protected]> 寫道： >> > Hi Kelly, >> > >> > the bigtop-toolchain needs more love for amazon-linux. There is already >> a JIRA for it : BIGTOP-1883 , but it should be improved slightly. >> > >> > Olaf >> > >> > >> > >> > > Am 09.06.2015 um 21:35 schrieb Kelly, Jonathan <[email protected]>: >> > > >> > > Oh, OK, do you guys want it in the toolchain for all distros by >> default? I >> > > wasn't sure if you did (I probably should have asked), so for now I >> only >> > > added it to AmazonLinux in our own repo. If you'd like to add it to >> all >> > > distros, that would be great. >> > > >> > > Thanks, >> > > Jonathan Kelly >> > > >> > > >> > > >> > > >> > > On 6/9/15, 11:56 AM, "Olaf Flebbe" <[email protected]> wrote: >> > > >> > >> Hi Evans, >> > >> >> > >> Just discovered that snappy-devel is missing from bigtop_toolchain. >> Will >> > >> open a JIRA for it. >> > >> >> > >> Olaf >> > >> >> > >> >> > >>> Am 09.06.2015 um 05:13 schrieb Evans Ye <[email protected]>: >> > >>> >> > >>> Yup, you need snappy-devel to be installed. Our new distribution >> > >>> targeting bigtop 1.0 is building hadoop with snappy, so the snappy >> > >>> compression should work at your side as well. :) >> > >>> >> > >>> 2015-06-09 6:57 GMT+08:00 Kelly, Jonathan <[email protected]>: >> > >>> Nvm, answered my own question. Apparently I want snappy-devel rather >> > >>> than snappy. >> > >>> >> > >>> ~ Jonathan Kelly >> > >>> >> > >>> From: <Kelly>, Jonathan Kelly <[email protected]> >> > >>> Date: Monday, June 8, 2015 at 3:31 PM >> > >>> To: "[email protected]" <[email protected]> >> > >>> Subject: Bundling Snappy with Hadoop >> > >>> >> > >>> Is anybody bundling Hadoop with Snappy support on Bigtop? I noticed >> > >>> that >> > >>> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/h >> > >>> adoop/do-component-build checks for the presence of >> > >>> /usr/lib/libsnappy.so or /usr/lib64/libsnappy.so and sets >> $BUNDLE_SNAPPY >> > >>> accordingly, so it seems that I should just need to install the >> snappy >> > >>> package, and Hadoop will automatically be built with Snappy support. >> > >>> However, when I install the snappy package ("sudo yum install -y >> snappy" >> > >>> on AmazonLinux 2015.03), the library file that gets installed is >> > >>> /usr/lib64/libsnappy.so, not /usr/lib64/libsnappy.so.1, so >> installing >> > >>> the snappy package has no effect on the Hadoop build. >> > >>> >> > >>> Thanks, >> > >>> Jonathan Kelly >> > >>> >> > >> >> > > >> > >> >> >
