[
https://issues.apache.org/jira/browse/SPARK-33144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798232#comment-17798232
]
Wan Kun commented on SPARK-33144:
---------------------------------
Speculative task also can cause this issue:
Spark Driver Log
23/12/17 11:28:10 INFO TaskSetManager: Starting task 39.1 in stage 5.0 (TID
154) (node-013.com, executor 18, partition 39, PROCESS_LOCAL, 7317 bytes)
taskResourceAssignments Map()
23/12/17 11:28:11 INFO TaskSetManager: Killing attempt 1 for task 39.1 in stage
5.0 (TID 154) on node-013.com as the attempt 0 succeeded on
hdc42-mcc10-01-0210-6008-022.com
23/12/17 11:28:11 INFO DAGScheduler: ResultStage 5 (main at
NativeMethodAccessorImpl.java:0) finished in 2.973 s
23/12/17 11:28:11 INFO DAGScheduler: Job 2 is finished. Cancelling potential
speculative or zombie tasks for this job
23/12/17 11:28:11 INFO YarnClusterScheduler: Killing all running tasks in stage
5: Stage finished
23/12/17 11:28:11 INFO DAGScheduler: Job 2 finished: main at
NativeMethodAccessorImpl.java:0, took 2.992123 s
23/12/17 11:28:11 INFO FileFormatWriter: Start to commit write Job
9797dcfc-fa17-47ce-907c-97932130f3d3.
23/12/17 11:28:11 INFO FileFormatWriter: Write Job
9797dcfc-fa17-47ce-907c-97932130f3d3 committed. Elapsed time: 56 ms.
//
// delete ["", "user", "hive",
"warehouse",".hive-staging_hive_2023-12-17_11-27-52_763_730376237774235462-1","-ext-10000","_temporary"]
// and create ["", "user", "hive",
"warehouse",".hive-staging_hive_2023-12-17_11-27-52_763_730376237774235462-1","-ext-10000","_SUCCESS"]
//
23/12/17 11:28:11 INFO FileFormatWriter: Finished processing stats for write
job 9797dcfc-fa17-47ce-907c-97932130f3d3.
23/12/17 11:28:12 WARN TaskSetManager: Lost task 39.1 in stage 5.0 (TID 154)
(node-013.com executor 18): TaskKilled (Stage finished)
23/12/17 11:28:12 INFO TaskSetManager: task 39.1 in stage 5.0 (TID 154) failed,
but the task will not be re-executed (either because the task failed with a
shuffle data fetch failure, so the previous stage needs to be re-run, or
because a different copy of the task has already succeeded).
23/12/17 11:28:12 INFO YarnClusterScheduler: Removed TaskSet 5.0, whose tasks
have all completed, from pool
...
23/12/17 11:28:14 INFO Hive: New loading path =
viewfs://nn/user/hive/warehouse/.hive-staging_hive_2023-12-17_11-27-52_763_730376237774235462-1/-ext-10000/_temporary/0
with partSpec \{week_beg_dt=, ind=}
...
23/12/17 11:28:14 ERROR Hive: Exception when loading partition with parameters
partPath=viewfs://nn/user/hive/warehouse/.hive-staging_hive_2023-12-17_11-27-52_763_730376237774235462-1/-ext-10000/_temporary/0,
table=gross_chrn_byr_srw, partSpec=\{week_beg_dt=, ind=}, replace=true,
listBucketingEnabled=false, isAcid=false, hasFollowingStatsTask=false
org.apache.hadoop.hive.ql.metadata.HiveException: get partition: Value for key
week_beg_dt is null or empty
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2240)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2188)
at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1618)
at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:1929)
at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:1920)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
task 39.1 in stage 5.0 (TID 154) log:
Although the task is killed, the parquet file is still created.
2023-12-17 11:28:13 /10.216.139.145 create ["", "user", "hive",
"warehouse",".hive-staging_hive_2023-12-17_11-27-52_763_730376237774235462-1","-ext-10000","_temporary","0","_temporary","attempt_202312171128088986155311074942191_0005_m_000039_154","week_beg_dt=2023-09-24","ind=T4","part-00039-cd9174aa-8010-41fa-8fbe-46d01d2b1efb.c000"]
Delete subdirectory $OUTPUT_DIR/_temporary/0/xxx and caused directory
$OUTPUT_DIR/_temporary/0 leak:
2023-12-17 11:28:13 /10.216.139.145 delete ["", "user", "hive",
"warehouse",".hive-staging_hive_2023-12-17_11-27-52_763_730376237774235462-1","-ext-10000","_temporary","0","_temporary","attempt_202312171128088986155311074942191_0005_m_000039_154"]
23/12/17 11:28:12 INFO ParquetRecordWriterWrapper: creating real writer to
write at
viewfs://nn/user/hive/warehouse/.hive-staging_hive_2023-12-17_11-27-52_763_730376237774235462-1/-ext-10000/_temporary/0/_temporary/attempt_202312171128088986155311074942191_0005_m_000039_154/week_beg_dt=2023-09-24/ind=T4/part-00039-cd9174aa-8010-41fa-8fbe-46d01d2b1efb.c000
23/12/17 11:28:12 ERROR Utils: Aborting task
org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.InterruptedIOException: Interrupted waiting to send RPC request to
server
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:274)
at
org.apache.spark.sql.hive.execution.HiveOutputWriter.<init>(HiveFileFormat.scala:131)
at
org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.newInstance(HiveFileFormat.scala:104)
at
org.apache.spark.sql.execution.datasources.DynamicPartitionDataWriter.newOutputWriter(FileFormatDataWriter.scala:268)
at
org.apache.spark.sql.execution.datasources.DynamicPartitionDataWriter.write(FileFormatDataWriter.scala:291)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:280)
at
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1499)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:288)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:211)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1465)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.InterruptedIOException: Interrupted waiting to send RPC
request to server
at org.apache.hadoop.ipc.Client.call(Client.java:1807)
at org.apache.hadoop.ipc.Client.call(Client.java:1795)
at org.apache.hadoop.ipc.Client.call(Client.java:1670)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:270)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:144)
at com.sun.proxy.$Proxy15.create(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:462)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:438)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:168)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:160)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:365)
at com.sun.proxy.$Proxy16.create(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:280)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1271)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1250)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1232)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1170)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:556)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:553)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:567)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:494)
at org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:197)
at
org.apache.hadoop.fs.viewfs.ChRootedFileSystem.create(ChRootedFileSystem.java:206)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.create(ViewFileSystem.java:486)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
at
org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)
at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:347)
at
org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:314)
at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:480)
at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:430)
at
org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:425)
at
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:70)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:137)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:126)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:286)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:271)
... 16 more
Caused by: java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1368)
at org.apache.hadoop.ipc.Client.call(Client.java:1802)
... 58 more
23/12/17 11:28:12 WARN SQLHadoopMapReduceCommitProtocol: Exception while
aborting attempt_202312171128088986155311074942191_0005_m_000039_154
java.io.InterruptedIOException: Interrupted waiting to send RPC request to
server
at org.apache.hadoop.ipc.Client.call(Client.java:1807)
at org.apache.hadoop.ipc.Client.call(Client.java:1795)
at org.apache.hadoop.ipc.Client.call(Client.java:1670)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:270)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:144)
at com.sun.proxy.$Proxy15.delete(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:902)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:438)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:168)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:160)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:365)
at com.sun.proxy.$Proxy16.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1662)
at
org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:992)
at
org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:989)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:999)
at org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:268)
at
org.apache.hadoop.fs.viewfs.ChRootedFileSystem.delete(ChRootedFileSystem.java:222)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.delete(ViewFileSystem.java:500)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.abortTask(FileOutputCommitter.java:648)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.abortTask(FileOutputCommitter.java:637)
at
org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.abortTask(HadoopMapReduceCommitProtocol.scala:353)
at
org.apache.spark.sql.execution.datasources.FileFormatDataWriter.abort(FileFormatDataWriter.scala:95)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$2(FileFormatWriter.scala:285)
at
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1510)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:288)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:211)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1465)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1368)
at org.apache.hadoop.ipc.Client.call(Client.java:1802)
... 40 more
> Connot insert overwite multiple partition, get exception "get partition:
> Value for key name is null or empty"
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-33144
> URL: https://issues.apache.org/jira/browse/SPARK-33144
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.1, 3.2.1
> Environment: hadoop 2.7.3 + spark 3.0.1
> hadoop 2.7.3 + spark 3.2.1
> Reporter: CHC
> Priority: Major
>
> When:
> {code:sql}
> create table tmp.spark_multi_partition(
> id int
> )
> partitioned by (name string, version string)
> stored as orc
> ;
> set hive.exec.dynamic.partition=true;
> set spark.hadoop.hive.exec.dynamic.partition=true;
>
> set hive.exec.dynamic.partition.mode=nonstrict;
> set spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict;
> insert overwrite table tmp.spark_multi_partition partition (name, version)
> select
> *
> from (
> select
> 1 as id
> , 'hadoop' as name
> , '2.7.3' as version
> union
> select
> 2 as id
> , 'spark' as name
> , '3.0.1' as version
> union
> select
> 3 as id
> , 'hive' as name
> , '2.3.4' as version
> ) as A;
> {code}
> and get exception:
> {code:bash}
> INFO load-dynamic-partitions-0 [hive.ql.metadata.Hive:1919]: New loading path
> =
> hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/.hive-staging_hive_2020-10-14_09-15-27_718_4118806337003279343-1/-ext-10000/name=spark/version=3.0.1
> with partSpec {name=spark, version=3.0.1}
> 20/10/14 09:15:33 INFO load-dynamic-partitions-1
> [hive.ql.metadata.Hive:1919]: New loading path =
> hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/.hive-staging_hive_2020-10-14_09-15-27_718_4118806337003279343-1/-ext-10000/name=hadoop/version=2.7.3
> with partSpec {name=hadoop, version=2.7.3}
> 20/10/14 09:15:33 INFO load-dynamic-partitions-2
> [hive.ql.metadata.Hive:1919]: New loading path =
> hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/.hive-staging_hive_2020-10-14_09-15-27_718_4118806337003279343-1/-ext-10000/name=hive/version=2.3.4
> with partSpec {name=hive, version=2.3.4}
> 20/10/14 09:15:33 INFO load-dynamic-partitions-3
> [hive.ql.metadata.Hive:1919]: New loading path =
> hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/.hive-staging_hive_2020-10-14_09-15-27_718_4118806337003279343-1/-ext-10000/_temporary/0
> with partSpec {name=, version=}
> 20/10/14 09:15:33 ERROR load-dynamic-partitions-3
> [hive.ql.metadata.Hive:1937]: Exception when loading partition with
> parameters
> partPath=hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/.hive-staging_hive_2020-10-14_09-15-27_718_4118806337003279343-1/-ext-10000/_temporary/0,
> table=spark_multi_partition, partSpec={name=, version=}, replace=true,
> listBucketingEnabled=false, isAcid=false, hasFollowingStatsTask=false
> org.apache.hadoop.hive.ql.metadata.HiveException: get partition: Value for
> key name is null or empty
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2233)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2181)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1611)
> at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:1922)
> at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:1913)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 20/10/14 09:15:33 INFO Delete-Thread-0
> [org.apache.hadoop.fs.TrashPolicyDefault:168]: Moved:
> 'hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/name=spark/version=3.0.1/part-00001-b745147b-600f-4c79-8ba2-12a99283b0a9.c000'
> to trash at:
> hdfs://namespace/user/hive/.Trash/Current/apps/hive/warehouse/tmp.db/spark_multi_partition/name=spark/version=3.0.1/part-00001-b745147b-600f-4c79-8ba2-12a99283b0a9.c000
> 20/10/14 09:15:33 INFO load-dynamic-partitions-0
> [org.apache.hadoop.hive.common.FileUtils:520]: Creating directory if it
> doesn't exist:
> hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/name=spark/version=3.0.1
> 20/10/14 09:15:33 INFO Delete-Thread-0
> [org.apache.hadoop.fs.TrashPolicyDefault:168]: Moved:
> 'hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/name=hive/version=2.3.4/part-00002-b745147b-600f-4c79-8ba2-12a99283b0a9.c000'
> to trash at:
> hdfs://namespace/user/hive/.Trash/Current/apps/hive/warehouse/tmp.db/spark_multi_partition/name=hive/version=2.3.4/part-00002-b745147b-600f-4c79-8ba2-12a99283b0a9.c000
> 20/10/14 09:15:33 INFO load-dynamic-partitions-2
> [org.apache.hadoop.hive.common.FileUtils:520]: Creating directory if it
> doesn't exist:
> hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/name=hive/version=2.3.4
> 20/10/14 09:15:33 INFO Delete-Thread-0
> [org.apache.hadoop.fs.TrashPolicyDefault:168]: Moved:
> 'hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/name=hadoop/version=2.7.3/part-00000-b745147b-600f-4c79-8ba2-12a99283b0a9.c000'
> to trash at:
> hdfs://namespace/user/hive/.Trash/Current/apps/hive/warehouse/tmp.db/spark_multi_partition/name=hadoop/version=2.7.3/part-00000-b745147b-600f-4c79-8ba2-12a99283b0a9.c000
> 20/10/14 09:15:33 INFO load-dynamic-partitions-1
> [org.apache.hadoop.hive.common.FileUtils:520]: Creating directory if it
> doesn't exist:
> hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/name=hadoop/version=2.7.3
> Error in query: org.apache.hadoop.hive.ql.metadata.HiveException: Exception
> when loading 4 in table spark_multi_partition with
> loadPath=hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/.hive-staging_hive_2020-10-14_09-15-27_718_4118806337003279343-1/-ext-10000;
> org.apache.spark.sql.AnalysisException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Exception when loading 4 in
> table spark_multi_partition with
> loadPath=hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/.hive-staging_hive_2020-10-14_09-15-27_718_4118806337003279343-1/-ext-10000;
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:113)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.loadDynamicPartitions(HiveExternalCatalog.scala:924)
> at
> org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.loadDynamicPartitions(ExternalCatalogWithListener.scala:189)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:258)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:102)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)
> at
> org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:120)
> at
> org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)
> at
> org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3618)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3616)
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:229)
> at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
> at
> org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:607)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:63)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:377)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:496)
> at scala.collection.Iterator.foreach(Iterator.scala:941)
> at scala.collection.Iterator.foreach$(Iterator.scala:941)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:490)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:282)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Exception when
> loading 4 in table spark_multi_partition with
> loadPath=hdfs://namespace/apps/hive/warehouse/tmp.db/spark_multi_partition/.hive-staging_hive_2020-10-14_09-15-27_718_4118806337003279343-1/-ext-10000
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:1963)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.sql.hive.client.Shim_v2_1.loadDynamicPartitions(HiveShim.scala:1226)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$loadDynamicPartitions$1(HiveClientImpl.scala:903)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:227)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:226)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:276)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.loadDynamicPartitions(HiveClientImpl.scala:894)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$loadDynamicPartitions$1(HiveExternalCatalog.scala:944)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:103)
> ... 47 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.ql.metadata.HiveException: get partition: Value for
> key name is null or empty
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:1954)
> ... 62 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: get partition:
> Value for key name is null or empty
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2233)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2181)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1611)
> at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:1922)
> at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:1913)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
