Michael Brown has uploaded a new patch set (#4).
Change subject: IMPALA-3864: qgen: reduce likelihood of create_query()
exceptions
......................................................................
IMPALA-3864: qgen: reduce likelihood of create_query() exceptions
1. Fix a bug in which the computation to produce the string for an
exception was raising a TypeError. We fix the bug by changing how the
string is built.
2. Fix a bug in which we tried to choose a relational function (defined
as taking in more than one argument and returning a Boolean) and were
looking for its weight in QueryProfile.weights.RELATIONAL_FUNCS, but
the function wasn't defined in that dictionary. We fix the bug by
defining weights for those functions.
3. Fix a bug in which QueryProfile.choose_func_signatures() was choosing
a function without taking into account the set of functions in the
signatures given to it. We fix the bug by pruning off the weights of
functions that aren't included in provided signatures. We also add a
note explaining how the weights defined are "best effort", since
sometimes functions will be pruned.
4. Add Char signatures to LessThan, GreaterThan, LessThanOrEquals,
GreaterThanOrEquals. Debugging #3 above brought this to my attention.
5. Make changes to aid in debugging or testing:
a. Add funcs.Signature representation.
b. Move the code in query_generator.__main__ to
generate_queries_for_manual_inspection(); call it from __main__.
c. Increase the number of fake columns when calling
generate_queries_for_manual_inspection(), which is useful for
testing.
d. Rename a few variables. For some reason in the query_generator
module there are a lot of overwritten variables, which makes
debugging difficult.
Testing:
1. impala-python tests/comparison/query_generator.py produces far fewer
exceptions. The ones for this bug are fixed, but see IMPALA-3890.
2. Full 3 and 6 hour runs of the query generator system don't show any
obvious regressions in behavior.
Change-Id: Idd9434a92973176aefb99e11e039209cac3cea65
---
M tests/comparison/funcs.py
M tests/comparison/query_generator.py
M tests/comparison/query_profile.py
3 files changed, 55 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/20/3720/4
--
To view, visit http://gerrit.cloudera.org:8080/3720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idd9434a92973176aefb99e11e039209cac3cea65
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <[email protected]>
Gerrit-Reviewer: David Knupp <[email protected]>
Gerrit-Reviewer: Michael Brown <[email protected]>
Gerrit-Reviewer: Taras Bobrovytsky <[email protected]>
