[
https://issues.apache.org/jira/browse/HIVE-29483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29483:
--------------------------------
Description:
Follow-up for HIVE-29470
{code:java}
implement an integration testing mechanism (inspired by current qtest drivers
maybe), that can utilize a real mini yarn cluster, that relies on the localized
hive-exec jar (and figure out first how the current TestMiniTezCliDriver works
and all tests pass on that driver with the above hack patch in place) <--
nice-to-have long-term
{code}
history recap, given this POC patch:
https://github.com/abstractdog/hive/commit/06ce23fb04bb2e862d0cf511f5ac214ce1e6bee9
on a classic yarn cluster, this patch makes all hive queries fail, given that
the hive-exec.jar (referred to as app jar) localization is skipped, so when a
tez job starts (in a yarn application), that jar is not present, so it
immediately fails with some kind of ClassNotFoundException kinda problem
the implemented testing here is definitely a heavy, integrated testing one, so
I can imagine that it won't included into the precommit phase, but users must
be able to start a pseudo-distributed hadoop/yarn/hdfs cluster and run tez jobs
against that, because apparently, the mini clusters won't cover some cases
was:
Follow-up for HIVE-29470
{code}
implement an integration testing mechanism (inspired by current qtest drivers
maybe), that can utilize a real mini yarn cluster, that relies on the localized
hive-exec jar (and figure out first how the current TestMiniTezCliDriver works
and all tests pass on that driver with the above hack patch in place) <--
nice-to-have long-term
{code}
> Implement an integration testing mechanism with real Tez on Yarn including
> resource localization
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-29483
> URL: https://issues.apache.org/jira/browse/HIVE-29483
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Priority: Major
>
> Follow-up for HIVE-29470
> {code:java}
> implement an integration testing mechanism (inspired by current qtest drivers
> maybe), that can utilize a real mini yarn cluster, that relies on the
> localized hive-exec jar (and figure out first how the current
> TestMiniTezCliDriver works and all tests pass on that driver with the above
> hack patch in place) <-- nice-to-have long-term
> {code}
> history recap, given this POC patch:
> https://github.com/abstractdog/hive/commit/06ce23fb04bb2e862d0cf511f5ac214ce1e6bee9
> on a classic yarn cluster, this patch makes all hive queries fail, given that
> the hive-exec.jar (referred to as app jar) localization is skipped, so when a
> tez job starts (in a yarn application), that jar is not present, so it
> immediately fails with some kind of ClassNotFoundException kinda problem
> the implemented testing here is definitely a heavy, integrated testing one,
> so I can imagine that it won't included into the precommit phase, but users
> must be able to start a pseudo-distributed hadoop/yarn/hdfs cluster and run
> tez jobs against that, because apparently, the mini clusters won't cover some
> cases
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
