[
https://issues.apache.org/jira/browse/HIVE-21072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743070#comment-16743070
]
Barnabas Maidics commented on HIVE-21072:
-----------------------------------------
I've also tried running partitioned CTAS and ran into the same error. As I saw,
if the execution engine is TEZ, it works perfectly (that is why partition_ctas
worked with TestMiniLlapLocalCliDriver).
Using MR, the NPE was thrown because Hive tried to create a Map-only merge job
(_GenMapRedUtils.createMRWorkForMergingFiles_), but the _tableInfo_ of the
_FileSinkDesc_ doesn't contain an entry with the key of "partition_columns" and
we try to call split on a null.
{code:java}
String[] partNames = properties.getProperty(
org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_PARTITION_COLUMNS)
.split("/");
{code}
A possible quick fix is to *set _hive.merge.mapfiles_ to _false_* (as a
default, it is true) so these steps will be skipped
(_GenMapRedUtils.__isMergeRequired_ will return _false_).
But maybe I miss something about this feature. [~jcamachorodriguez] what do you
think the long-term fix would be for this?
> NPE when running partitioned CTAS statements
> --------------------------------------------
>
> Key: HIVE-21072
> URL: https://issues.apache.org/jira/browse/HIVE-21072
> Project: Hive
> Issue Type: Bug
> Reporter: Liang-Chi Hsieh
> Priority: Major
> Labels: pull-request-available
>
> HIVE-20241 adds support of partitioned CTAS statements:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;{code}
>
> However, I've tried this feature by checking out latest branch-3, and
> encountered NPE:
> {code:java}
> hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
> FAILED: NullPointerException null
> {code}
> I also ran the query test partition_ctas.q. The test passes when using
> TestMiniLlapLocalCliDriver, but when I go to test it with TestCliDriver
> manually, it also throws NullPointerException:
> {code:java}
> 2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
> at
> org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
> at
> org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
