GitHub user ericl reopened a pull request:
https://github.com/apache/spark/pull/15725
[SPARK-18167] [DO NOT MERGE] Print out spark confs, and hive confs when
SQLQuerySuite fails
## What changes were proposed in this pull request?
It seems the proximate cause of the test failures is that `cast(str as
decimal)` in derby will raise an exception instead of returning NULL. This is a
problem since Hive sometimes inserts `__HIVE_DEFAULT_PARTITION__` entries into
the partition table as documented here:
https://github.com/apache/hive/blob/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1034
Basically, when these special default partitions are present, partition
pruning pushdown using the SQL-direct mode will fail due this cast exception.
As commented on in `MetaStoreDirectSql.java` above, this is normally fine since
Hive falls back to JDO pruning, however when the pruning predicate contains an
unsupported operator such as `>`, that will fail as well.
The only remaining question is why this behavior is nondeterministic. We
know that when the test flakes, retries do not help, therefore the cause must
be environmental. The current best hypothesis is that some config is different
between different jenkins runs, which is why this PR prints out the Spark SQL
and Hive confs for the test. The hope is that by comparing the config state for
failure vs success we can isolate the root cause of the flakiness.
## How was this patch tested?
I verified the confs are printed out in a readable form locally.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ericl/spark print-confs-out
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15725.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15725
----
commit 88adf1d41bce4b2eb1e8c255de3008800f6913a7
Author: Eric Liang <[email protected]>
Date: 2016-11-02T01:38:57Z
Tue Nov 1 18:38:57 PDT 2016
commit 0dca3ac4dec6efbc3a52a6c995bc43296358a449
Author: Eric Liang <[email protected]>
Date: 2016-11-02T01:49:24Z
Merge branch 'master' into print-confs-out
commit 5129d9dcd3a016f5e88d0855b024b5ce2d20e354
Author: Eric Liang <[email protected]>
Date: 2016-11-02T18:25:46Z
Wed Nov 2 11:25:46 PDT 2016
commit 5f7d7996dea9a04af82959b3c55ba6c8a7f33a50
Author: Eric Liang <[email protected]>
Date: 2016-11-02T18:26:09Z
Wed Nov 2 11:26:09 PDT 2016
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
