[
https://issues.apache.org/jira/browse/SPARK-41684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681657#comment-17681657
]
xuanzhiang commented on SPARK-41684:
------------------------------------
u can set spark.sql.hive.convertMetastoreParquet=false .
> spark3 read the one partition data and write to anthor partition cause error
> ----------------------------------------------------------------------------
>
> Key: SPARK-41684
> URL: https://issues.apache.org/jira/browse/SPARK-41684
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.2.2
> Reporter: sinlang
> Priority: Major
>
> spark3 read the one partition data and write to anthor partition cause error
> {code:java}
> 1 create temporary view t1 :
> select * from jt_ods.ods_ebi_stm_retail_settle_detail_full_di
> where dt = '2022-12-21'
> union all (
> select * from jt_ods.ods_ebi_stm_retail_settle_detail_full_df as i
> where i.dt = '2022-12-20'
> and not exists(select 1 from jt_ods.ods_ebi_stm_retail_settle_detail_full_di
> as d where d.dt = '2022-12-21' and i.id = d.id))
> 2 insert data :
> insert sql insert overwrite table
> jt_ods.ods_ebi_stm_retail_settle_detail_full_df partition(dt = '2022-12-21')
> select * from t distribute by rand() {code}
> {code:java}
> 2022-12-22 16:29:48 Driver ERROR
> org.apache.spark.deploy.yarn.ApplicationMaster
> User class threw exception: org.apache.spark.sql.AnalysisException: Cannot
> overwrite a path that is also being read from.
> org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also
> being read from.
> at
> org.apache.spark.sql.errors.QueryCompilationErrors$.cannotOverwritePathBeingReadFromError(QueryCompilationErrors.scala:1834)
> at
> org.apache.spark.sql.execution.command.DDLUtils$.verifyNotReadPath(ddl.scala:980)
> at
> org.apache.spark.sql.execution.datasources.DataSourceAnalysis$$anonfun$apply$1.applyOrElse(DataSourceStrategy.scala:221)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
