What have you tried? Did you try building with the old Cloudera Impala repo at https://github.com/cloudera/impala/tree/cdh5-2.3.0_5.5.4?
On Wed, Oct 5, 2016 at 10:48 AM, Pradeep Nayak <[email protected]> wrote: > Hi - I have verified my changes in the dev cluster which I had. However the > test/staging cluster is at 5.5.4. Is there a quick way to build it for that > version. ? > > Cheers! > Pradeep > > http://pradeepnayak.in > http://twitter.com/_prdp > > On Tue, Sep 13, 2016 at 4:54 PM, Tim Armstrong <[email protected]> > wrote: > >> It looks like you built the debug Impala with dynamic linking >> (-build_shared_libs or -so) whereas the CDH build uses static linking - >> that might explain why the release build worked ok. >> >> It's expected that your binary is much larger, in the CDH packages the >> debug symbols were stripped out and put in a separate impalad.debug file, >> whereas just building a dev environment doesn't do that step. Should work >> fine though. >> >> On Tue, Sep 13, 2016 at 4:44 PM, Pradeep Nayak <[email protected]> >> wrote: >> >>> Well when I built the debug one and replaced it my cluster it seemed to >>> have more dependencies which it was not able to find; >>> >>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >>> 5.8.0.p0.42/lib/impala/sbin-debug$ ldd impalad >>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version >>> `GLIBCXX_3.4.20' not found (required by ./impalad) >>> linux-vdso.so.1 => (0x00007ffd87dc6000) >>> libjsig.so => not found >>> libTestUtil.so => not found >>> libUtil.so => not found >>> libRuntime.so => not found >>> libExec.so => not found >>> libCodeGen.so => not found >>> libExprs.so => not found >>> libRpc.so => not found >>> libService.so => not found >>> libStatestore.so => not found >>> libCatalog.so => not found >>> libResourceBroker.so => not found >>> libImpalaThrift.so => not found >>> libGlobalFlags.so => not found >>> libCommon.so => not found >>> libUdf.so => not found >>> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 >>> (0x00007fd4661be000) >>> libThriftSaslTransport.so => not found >>> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 >>> (0x00007fd465f5f000) >>> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >>> (0x00007fd465b83000) >>> libgutil.so => not found >>> libhdfs.so.0.0.0 => not found >>> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd46596a000) >>> libjvm.so => not found >>> libkudu_client.so.0 => not found >>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 >>> (0x00007fd465762000) >>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 >>> (0x00007fd46555e000) >>> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 >>> (0x00007fd46525a000) >>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd464f54000) >>> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 >>> (0x00007fd464d3e000) >>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 >>> (0x00007fd464b20000) >>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd46475b000) >>> /lib64/ld-linux-x86-64.so.2 (0x00007fd4663d9000) >>> >>> >>> >>> I went ahead and build the retail one and that was OKAY. >>> >>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >>> 5.8.0.p0.42/lib/impala/sbin-retail$ ldd impalad >>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version >>> `CXXABI_1.3.8' not found (required by ./impalad) >>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version >>> `GLIBCXX_3.4.20' not found (required by ./impalad) >>> linux-vdso.so.1 => (0x00007fffc2679000) >>> libjsig.so => not found >>> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 >>> (0x00007f153629d000) >>> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 >>> (0x00007f153603e000) >>> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >>> (0x00007f1535c62000) >>> libjvm.so => not found >>> libkudu_client.so.0 => not found >>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 >>> (0x00007f1535a5a000) >>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 >>> (0x00007f1535856000) >>> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 >>> (0x00007f1535552000) >>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f153524c000) >>> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 >>> (0x00007f1535036000) >>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 >>> (0x00007f1534e18000) >>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1534a53000) >>> /lib64/ld-linux-x86-64.so.2 (0x00007f15364b8000) >>> >>> >>> However I see the sizes being different. The one which came with cloudera >>> parcels was of 37M and one which I built was 384MB!! Why is that ? >>> >>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh impalad >>> 348M impalad >>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/impala.retail.orig >>> 37M /home/ubuntu/impala.retail.orig >>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh libfesupport.so >>> 346M libfesupport.so >>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/libfesupport.so.retail.orig >>> 39M /home/ubuntu/libfesupport.so.retail.orig >>> >>> >>> To conclude I am able to verify the changes which I made in the retail >>> build but not with the debug build. We can probably document what needs to >>> be replaced to test changes in a real cluster with someone's custom impala >>> binaries. >>> >>> 1. impalad in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impa >>> la/sbin-retail >>> 2. libfesupport.so in /opt/cloudera/parcels/CDH-5 >>> .8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail >>> 3. impala-frontend-0.1-SNAPSHOT.jar in >>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/. >>> The impala-frontend-0.1-SNAPSHOT.jar file in >>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/lib/ >>> symlinks to the one in /opt/cloudera/parcels/CDH-5.8. >>> 0-1.cdh5.8.0.p0.42/jars/. >>> >>> This is all which is need to test your changes. >>> >>> Regards, >>> Pradeep Nayak >>> >>> http://pradeepnayak.in >>> http://twitter.com/_prdp >>> >>> On Tue, Sep 13, 2016 at 4:13 PM, Pradeep Nayak <[email protected]> >>> wrote: >>> >>>> Okay thanks Tim. I tried that. >>>> >>>> I am seeing the below error, once I repalced the impalad and >>>> libfesupport.so. Same errors for statestore and catalogd. >>>> >>>> + true >>>> + export >>>> IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug >>>> + >>>> IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug >>>> + [[ true = '' ]] >>>> + '[' impalad = impalad ']' >>>> + exec >>>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/../../bin/impalad >>>> >>>> --flagfile=/run/cloudera-scm-agent/process/92-impala-IMPALAD/impala-conf/impalad_flags >>>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug/impalad: >>>> error while loading shared libraries: libTestUtil.so: cannot open shared >>>> object file: No such file or directory >>>> >>>> >>>> Regards, >>>> Pradeep Nayak >>>> >>>> http://pradeepnayak.in >>>> http://twitter.com/_prdp >>>> >>>> On Tue, Sep 13, 2016 at 2:16 PM, Tim Armstrong <[email protected]> >>>> wrote: >>>> >>>>> If the statestored and catalogd symlinks point to the new impalad >>>>> binary then you're good. >>>>> >>>>> On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <[email protected]> >>>>> wrote: >>>>> >>>>>> My confusion should I update the bin location or the sbin location >>>>>> here : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd >>>>>> and statestored appear to be symlinks to impalad. >>>>>> >>>>>> Regards, >>>>>> Pradeep Nayak >>>>>> >>>>>> http://pradeepnayak.in >>>>>> http://twitter.com/_prdp >>>>>> >>>>>> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5 >>>>>>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built. >>>>>>> How about catalogd and statestored ? >>>>>>> >>>>>>> Regards, >>>>>>> Pradeep Nayak >>>>>>> >>>>>>> http://pradeepnayak.in >>>>>>> http://twitter.com/_prdp >>>>>>> >>>>>>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Correct, impala-data-source-api*.jar rarely changes, but it will when >>>>>>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/ >>>>>>>> >>>>>>>> If you go through this process again with a build after that change >>>>>>>> goes in, then you'll need to update that jar. >>>>>>>> >>>>>>>> You can build it by running mvn package in Impala/ext-data-source. >>>>>>>> >>>>>>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong < >>>>>>>> [email protected]> wrote: >>>>>>>> > It looks like impala-data-source* may not be built in all case. It >>>>>>>> doesn't >>>>>>>> > really change much if at all so I doubt you will need to update it >>>>>>>> anyway. >>>>>>>> > >>>>>>>> > P.S. I agree with Jim, this is exactly the kind of question this >>>>>>>> list is >>>>>>>> > for. >>>>>>>> > >>>>>>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak < >>>>>>>> [email protected]> >>>>>>>> > wrote: >>>>>>>> > >>>>>>>> >> So here is what I understand: >>>>>>>> >> >>>>>>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/ >>>>>>>> >> build/debug/service >>>>>>>> >> >>>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l >>>>>>>> >> total 330928 >>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test >>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad* >>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so* >>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so* >>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 >>>>>>>> query-options-test >>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 >>>>>>>> session-expiry-test >>>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ >>>>>>>> >> >>>>>>>> >> and replace it with one on my test cluster for backend part. >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> For the frontend, i need to pick the below file: >>>>>>>> >> >>>>>>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar * >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> I couldn't find the impala-data-source-api*.jar file >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Regards, >>>>>>>> >> Pradeep Nayak >>>>>>>> >> >>>>>>>> >> http://pradeepnayak.in >>>>>>>> >> http://twitter.com/_prdp >>>>>>>> >> >>>>>>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong < >>>>>>>> [email protected]> >>>>>>>> >> wrote: >>>>>>>> >> >>>>>>>> >>> I don't think this is documented anywhere. You would need to >>>>>>>> replace the >>>>>>>> >>> impalad binary and libfesupport.so under sbin-retail or >>>>>>>> sbin-debug for the >>>>>>>> >>> release or debug builds respectively. You also need to replace >>>>>>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar. >>>>>>>> >>> >>>>>>>> >>> You also need to make sure the statestored and catalogd symlinks >>>>>>>> point to >>>>>>>> >>> the impalad binary, >>>>>>>> >>> >>>>>>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak < >>>>>>>> [email protected]> >>>>>>>> >>> wrote: >>>>>>>> >>> >>>>>>>> >>>> This question may sound noob, but apologies in advance. >>>>>>>> >>>> >>>>>>>> >>>> I have changes in the impala source code (both backend and >>>>>>>> frontend >>>>>>>> >>>> components). How do I go about installing this version of >>>>>>>> Impala on my >>>>>>>> >>>> cloudera test cluster ? I know the path where I can replace the >>>>>>>> front end >>>>>>>> >>>> built jar file, but not sure of the backend parts. >>>>>>>> >>>> >>>>>>>> >>>> Regards, >>>>>>>> >>>> Pradeep Nayak >>>>>>>> >>>> >>>>>>>> >>>> http://pradeepnayak.in >>>>>>>> >>>> http://twitter.com/_prdp >>>>>>>> >>>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
