[
https://issues.apache.org/jira/browse/SPARK-50668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kevinshin updated SPARK-50668:
------------------------------
Description:
I have some SQL scripts for pushing the calculation results of the data
warehouse to the JDBC DATASOURCE.
When I attempted to upgrade from Spark 3.3.3 to Spark 3.5.4, I found that one
of them failed to run properly: it pushed data to the RDS database of Alibaba
Cloud, but other similar scripts worked without any issues.
This script can run normally on both Spark 3.3.3 and 3.4.4.
SCRIPT:
{code:java}
CREATE TEMPORARY VIEW test
USING org.apache.spark.sql.jdbc
OPTIONS (
url
"jdbc:mysql://rm-abc.mysql.rds.aliyuncs.com:3306/db_01?useUnicode=true&characterEncoding=UTF-8&useSSL=false",
dbtable "`apdata`",
user 'sgroot',
password '####'
);{code}
{code:java}
desc sg_apdata_v_prd;
user_name string
region string
ap_name string
dtime string
ddate string{code}
{code:java}
explain
INSERT INTO sg_apdata_v_prd
select 'name' as user_name, 'region' as region, 'ap_name' as ap_name,
'2024-12-24 23:45:00' as dtime, '2024-12-24' as ddate;{code}
{code:java}
== Physical Plan ==
org.apache.spark.sql.AnalysisException: [UNSUPPORTED_INSERT.RDD_BASED] Can't
insert into the target. An RDD-based table is not allowed.;
'InsertIntoStatement Project [user_name#0, region#1, ap_name#2, dtime#3,
staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils,
StringType, readSidePadding, ddate#4, 10, true, false, true) AS ddate#5],
false, false, false
+- Project [name AS user_name#67, region AS region#68, ap_name AS
ap_name#69, 2024-12-24 23:45:00 AS dtime#70, 2024-12-24 AS ddate#71]
+- OneRowRelation {code}
was:
I have some SQL scripts for pushing the calculation results of the data
warehouse to the JDBC DATASOURCE.
When I attempted to upgrade from Spark 3.3.3 to Spark 3.5.4, I found that one
of them failed to run properly: it pushed data to the RDS database of Alibaba
Cloud, but other similar scripts worked without any issues.
This script can run normally on both Spark 3.3.3 and 3.4.4.
SCRIPT:
{code:java}
CREATE TEMPORARY VIEW test
USING org.apache.spark.sql.jdbc
OPTIONS (
url
"jdbc:mysql://rm-abc.mysql.rds.aliyuncs.com:3306/db_01?useUnicode=true&characterEncoding=UTF-8&useSSL=false",
dbtable "`apdata`",
user 'sgroot',
password '####'
);{code}
{code:java}
desc sg_apdata_v_prd;
user_name string
region string
ap_name string
dtime string
ddate string{code}
{code:java}
explain
INSERT INTO sg_apdata_v_prd
select 'name' as user_name, 'region' as region, 'ap_name' as ap_name,
'2024-12-24 23:45:00' as dtime, '2024-12-24' as ddate;{code}
{code:java}
== Physical Plan ==
org.apache.spark.sql.AnalysisException: [UNSUPPORTED_INSERT.RDD_BASED] Can't
insert into the target. An RDD-based table is not allowed.;
'InsertIntoStatement Project [user_name#0, region#1, ap_name#2, dtime#3,
staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils,
StringType, readSidePadding, ddate#4, 10, true, false, true) AS ddate#5],
false, false, false
+- Project [name AS user_name#67, region AS region#68, ap_name AS
ap_name#69, 2024-12-24 23:45:00 AS dtime#70, 2024-12-24 AS ddate#71]
+- OneRowRelation {code}
> UNSUPPORTED_INSERT.RDD_BASED when insert into a jdbc datasource
> ---------------------------------------------------------------
>
> Key: SPARK-50668
> URL: https://issues.apache.org/jira/browse/SPARK-50668
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.4
> Environment: hdp 3.1.4
> spark 3.5.4
> Reporter: kevinshin
> Priority: Major
>
> I have some SQL scripts for pushing the calculation results of the data
> warehouse to the JDBC DATASOURCE.
> When I attempted to upgrade from Spark 3.3.3 to Spark 3.5.4, I found that one
> of them failed to run properly: it pushed data to the RDS database of Alibaba
> Cloud, but other similar scripts worked without any issues.
> This script can run normally on both Spark 3.3.3 and 3.4.4.
>
> SCRIPT:
> {code:java}
> CREATE TEMPORARY VIEW test
> USING org.apache.spark.sql.jdbc
> OPTIONS (
> url
> "jdbc:mysql://rm-abc.mysql.rds.aliyuncs.com:3306/db_01?useUnicode=true&characterEncoding=UTF-8&useSSL=false",
> dbtable "`apdata`",
> user 'sgroot',
> password '####'
> );{code}
>
> {code:java}
> desc sg_apdata_v_prd;
> user_name string
> region string
> ap_name string
> dtime string
> ddate string{code}
>
> {code:java}
> explain
> INSERT INTO sg_apdata_v_prd
> select 'name' as user_name, 'region' as region, 'ap_name' as ap_name,
> '2024-12-24 23:45:00' as dtime, '2024-12-24' as ddate;{code}
>
> {code:java}
> == Physical Plan ==
>
>
> org.apache.spark.sql.AnalysisException: [UNSUPPORTED_INSERT.RDD_BASED] Can't
> insert into the target. An RDD-based table is not allowed.;
>
> 'InsertIntoStatement Project [user_name#0, region#1, ap_name#2, dtime#3,
> staticinvoke(class
> org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType,
> readSidePadding, ddate#4, 10, true, false, true) AS ddate#5], false, false,
> false
>
> +-
> Project [name AS user_name#67, region AS region#68, ap_name AS ap_name#69,
> 2024-12-24 23:45:00 AS dtime#70, 2024-12-24 AS ddate#71]
>
> +- OneRowRelation {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
