Hi - I have verified my changes in the dev cluster which I had. However the test/staging cluster is at 5.5.4. Is there a quick way to build it for that version. ?
Cheers! Pradeep http://pradeepnayak.in http://twitter.com/_prdp On Tue, Sep 13, 2016 at 4:54 PM, Tim Armstrong <[email protected]> wrote: > It looks like you built the debug Impala with dynamic linking > (-build_shared_libs or -so) whereas the CDH build uses static linking - > that might explain why the release build worked ok. > > It's expected that your binary is much larger, in the CDH packages the > debug symbols were stripped out and put in a separate impalad.debug file, > whereas just building a dev environment doesn't do that step. Should work > fine though. > > On Tue, Sep 13, 2016 at 4:44 PM, Pradeep Nayak <[email protected]> > wrote: > >> Well when I built the debug one and replaced it my cluster it seemed to >> have more dependencies which it was not able to find; >> >> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >> 5.8.0.p0.42/lib/impala/sbin-debug$ ldd impalad >> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version >> `GLIBCXX_3.4.20' not found (required by ./impalad) >> linux-vdso.so.1 => (0x00007ffd87dc6000) >> libjsig.so => not found >> libTestUtil.so => not found >> libUtil.so => not found >> libRuntime.so => not found >> libExec.so => not found >> libCodeGen.so => not found >> libExprs.so => not found >> libRpc.so => not found >> libService.so => not found >> libStatestore.so => not found >> libCatalog.so => not found >> libResourceBroker.so => not found >> libImpalaThrift.so => not found >> libGlobalFlags.so => not found >> libCommon.so => not found >> libUdf.so => not found >> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 >> (0x00007fd4661be000) >> libThriftSaslTransport.so => not found >> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 >> (0x00007fd465f5f000) >> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >> (0x00007fd465b83000) >> libgutil.so => not found >> libhdfs.so.0.0.0 => not found >> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd46596a000) >> libjvm.so => not found >> libkudu_client.so.0 => not found >> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 >> (0x00007fd465762000) >> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 >> (0x00007fd46555e000) >> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 >> (0x00007fd46525a000) >> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd464f54000) >> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 >> (0x00007fd464d3e000) >> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 >> (0x00007fd464b20000) >> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd46475b000) >> /lib64/ld-linux-x86-64.so.2 (0x00007fd4663d9000) >> >> >> >> I went ahead and build the retail one and that was OKAY. >> >> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >> 5.8.0.p0.42/lib/impala/sbin-retail$ ldd impalad >> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version >> `CXXABI_1.3.8' not found (required by ./impalad) >> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version >> `GLIBCXX_3.4.20' not found (required by ./impalad) >> linux-vdso.so.1 => (0x00007fffc2679000) >> libjsig.so => not found >> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 >> (0x00007f153629d000) >> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 >> (0x00007f153603e000) >> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >> (0x00007f1535c62000) >> libjvm.so => not found >> libkudu_client.so.0 => not found >> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 >> (0x00007f1535a5a000) >> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 >> (0x00007f1535856000) >> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 >> (0x00007f1535552000) >> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f153524c000) >> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 >> (0x00007f1535036000) >> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 >> (0x00007f1534e18000) >> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1534a53000) >> /lib64/ld-linux-x86-64.so.2 (0x00007f15364b8000) >> >> >> However I see the sizes being different. The one which came with cloudera >> parcels was of 37M and one which I built was 384MB!! Why is that ? >> >> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh impalad >> 348M impalad >> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/impala.retail.orig >> 37M /home/ubuntu/impala.retail.orig >> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh libfesupport.so >> 346M libfesupport.so >> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh >> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/libfesupport.so.retail.orig >> 39M /home/ubuntu/libfesupport.so.retail.orig >> >> >> To conclude I am able to verify the changes which I made in the retail >> build but not with the debug build. We can probably document what needs to >> be replaced to test changes in a real cluster with someone's custom impala >> binaries. >> >> 1. impalad in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impa >> la/sbin-retail >> 2. libfesupport.so in /opt/cloudera/parcels/CDH-5 >> .8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail >> 3. impala-frontend-0.1-SNAPSHOT.jar in >> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/. >> The impala-frontend-0.1-SNAPSHOT.jar file in >> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/lib/ >> symlinks to the one in /opt/cloudera/parcels/CDH-5.8. >> 0-1.cdh5.8.0.p0.42/jars/. >> >> This is all which is need to test your changes. >> >> Regards, >> Pradeep Nayak >> >> http://pradeepnayak.in >> http://twitter.com/_prdp >> >> On Tue, Sep 13, 2016 at 4:13 PM, Pradeep Nayak <[email protected]> >> wrote: >> >>> Okay thanks Tim. I tried that. >>> >>> I am seeing the below error, once I repalced the impalad and >>> libfesupport.so. Same errors for statestore and catalogd. >>> >>> + true >>> + export >>> IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug >>> + >>> IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug >>> + [[ true = '' ]] >>> + '[' impalad = impalad ']' >>> + exec >>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/../../bin/impalad >>> >>> --flagfile=/run/cloudera-scm-agent/process/92-impala-IMPALAD/impala-conf/impalad_flags >>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug/impalad: >>> error while loading shared libraries: libTestUtil.so: cannot open shared >>> object file: No such file or directory >>> >>> >>> Regards, >>> Pradeep Nayak >>> >>> http://pradeepnayak.in >>> http://twitter.com/_prdp >>> >>> On Tue, Sep 13, 2016 at 2:16 PM, Tim Armstrong <[email protected]> >>> wrote: >>> >>>> If the statestored and catalogd symlinks point to the new impalad >>>> binary then you're good. >>>> >>>> On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <[email protected]> >>>> wrote: >>>> >>>>> My confusion should I update the bin location or the sbin location >>>>> here : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd >>>>> and statestored appear to be symlinks to impalad. >>>>> >>>>> Regards, >>>>> Pradeep Nayak >>>>> >>>>> http://pradeepnayak.in >>>>> http://twitter.com/_prdp >>>>> >>>>> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <[email protected]> >>>>> wrote: >>>>> >>>>>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5 >>>>>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built. >>>>>> How about catalogd and statestored ? >>>>>> >>>>>> Regards, >>>>>> Pradeep Nayak >>>>>> >>>>>> http://pradeepnayak.in >>>>>> http://twitter.com/_prdp >>>>>> >>>>>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Correct, impala-data-source-api*.jar rarely changes, but it will when >>>>>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/ >>>>>>> >>>>>>> If you go through this process again with a build after that change >>>>>>> goes in, then you'll need to update that jar. >>>>>>> >>>>>>> You can build it by running mvn package in Impala/ext-data-source. >>>>>>> >>>>>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong < >>>>>>> [email protected]> wrote: >>>>>>> > It looks like impala-data-source* may not be built in all case. It >>>>>>> doesn't >>>>>>> > really change much if at all so I doubt you will need to update it >>>>>>> anyway. >>>>>>> > >>>>>>> > P.S. I agree with Jim, this is exactly the kind of question this >>>>>>> list is >>>>>>> > for. >>>>>>> > >>>>>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak < >>>>>>> [email protected]> >>>>>>> > wrote: >>>>>>> > >>>>>>> >> So here is what I understand: >>>>>>> >> >>>>>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/ >>>>>>> >> build/debug/service >>>>>>> >> >>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l >>>>>>> >> total 330928 >>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test >>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad* >>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so* >>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so* >>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 >>>>>>> query-options-test >>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 >>>>>>> session-expiry-test >>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ >>>>>>> >> >>>>>>> >> and replace it with one on my test cluster for backend part. >>>>>>> >> >>>>>>> >> >>>>>>> >> For the frontend, i need to pick the below file: >>>>>>> >> >>>>>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar * >>>>>>> >> >>>>>>> >> >>>>>>> >> I couldn't find the impala-data-source-api*.jar file >>>>>>> >> >>>>>>> >> >>>>>>> >> Regards, >>>>>>> >> Pradeep Nayak >>>>>>> >> >>>>>>> >> http://pradeepnayak.in >>>>>>> >> http://twitter.com/_prdp >>>>>>> >> >>>>>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong < >>>>>>> [email protected]> >>>>>>> >> wrote: >>>>>>> >> >>>>>>> >>> I don't think this is documented anywhere. You would need to >>>>>>> replace the >>>>>>> >>> impalad binary and libfesupport.so under sbin-retail or >>>>>>> sbin-debug for the >>>>>>> >>> release or debug builds respectively. You also need to replace >>>>>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar. >>>>>>> >>> >>>>>>> >>> You also need to make sure the statestored and catalogd symlinks >>>>>>> point to >>>>>>> >>> the impalad binary, >>>>>>> >>> >>>>>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak < >>>>>>> [email protected]> >>>>>>> >>> wrote: >>>>>>> >>> >>>>>>> >>>> This question may sound noob, but apologies in advance. >>>>>>> >>>> >>>>>>> >>>> I have changes in the impala source code (both backend and >>>>>>> frontend >>>>>>> >>>> components). How do I go about installing this version of >>>>>>> Impala on my >>>>>>> >>>> cloudera test cluster ? I know the path where I can replace the >>>>>>> front end >>>>>>> >>>> built jar file, but not sure of the backend parts. >>>>>>> >>>> >>>>>>> >>>> Regards, >>>>>>> >>>> Pradeep Nayak >>>>>>> >>>> >>>>>>> >>>> http://pradeepnayak.in >>>>>>> >>>> http://twitter.com/_prdp >>>>>>> >>>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
