David Knupp has posted comments on this change. Change subject: IMPALA-3898: Add a pytest skipif decorator based on presence of Impala LZO. ......................................................................
Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/3782/1//COMMIT_MSG Commit Message: Line 11: if Impala LZO isn't present, we want to skip tests that would otherwise > How did you test? [Not sure if you're asking how I test for the presence of Impala LZO, or how I generally tested the patch itself, so I'll answer both.] The buildall.sh scripts determines whether or not to build Impala LZO by checking to see if ${IMPALA_LZO} exists, so I also test for the presence of Impala LZO by determining whether ${IMPALA_LZO} is a valid directory, and also whether ${IMPALA_LZO}/build/libimpalalzo.so is a valid file. I identified broken tests and then tested the submitted patch by rebuilding Impala LZO with and without Impala LZO and running all the tests (though admittedly not in exhaustive mode) to see which, if any, failed. http://gerrit.cloudera.org:8080/#/c/3782/1/tests/common/skip.py File tests/common/skip.py: PS1, Line 119: def > Does this need to be a function in the class? Why not a top-level function? I scoped this inside the class because it seemed to be very specific to this application, not something that's generally useful. That said, I didn't know about the file that Michael pointed out, so I'll look into moving it there. http://gerrit.cloudera.org:8080/#/c/3782/1/tests/data_errors/test_data_errors.py File tests/data_errors/test_data_errors.py: PS1, Line 40: @SkipIfEnvironment.no_impalalzo > I agree with this - I think we can probably handle most cases by changing t So -- this appears to the case, but perhaps you could correct me, since I'm still shaky on how the test dimensions/vectors/matrices work. Even when the file format is 'text' and compression_codec is ' none', the test still fails due to an exception that gets raised by Beeswax client: E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted: E Unable to load libimpalalzo.so E dlerror: libimpalalzo.so: cannot open shared object file: No such file or directory E Error loading impala-lzo library. Check that the impala-lzo library is at version 2.7.0-cdh5-INTERNAL E E E Unable to load libimpalalzo.so E dlerror: libimpalalzo.so: cannot open shared object file: No such file or directory which itself arises when the test issues this query: select count(*) from functional_text_lzo.bad_text_lzo; which in turn references a table that is created during data load. I'm currently looking into what happens when we don't set up that table during data load if Impala LZO is not present, but I presume this will just cause the test to fail for other reasons. Line 46: # Only run on delimited text with no compression. > This comment seems to imply that this isn't run with LZO - is the comment s I think it might be. I found that test dimensions include vectors (am I using that term correctly?) where compression_codec is set to gzip, bzip, and snappy . I'll remove it. -- To view, visit http://gerrit.cloudera.org:8080/3782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: If61a7799205cd00d440196303a42db32c522f5b1 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: David Knupp <[email protected]> Gerrit-Reviewer: David Knupp <[email protected]> Gerrit-Reviewer: Jim Apple <[email protected]> Gerrit-Reviewer: Michael Brown <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-HasComments: Yes
