luckyliush commented on issue #6830:
URL: https://github.com/apache/seatunnel/issues/6830#issuecomment-2105600558
> > > try to add `rewriteBatchedStatements=true` parameter to your jdbc url
> >
> >
> > Thank you, but this parameter has been added beforeļ¼did not meet
expectations
>
> ```
> plugin_name = jdbc
> user = xxxxx
> url =
"jdbc:mysql://xxxxxx/xxxxxxx?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true&useSSL=false"
> enable_upsert = true
> generate_sink_sql = true
> database = db_name
> table = table_name
> primary_keys = [xxx,xxx]
> ```
>
> try with this config, it will auto generate insert sql, i use this config,
the write speed is good
`env {
execution.parallelism = 10
job.mode = "BATCH"
}
source {
Hive {
table_name = ""
metastore_uri = ""
result_table_name = "Table_test"
hdfs_site_path = "/home/hadoop/hadoop-3.2.2/etc/hadoop/hdfs-site.xml"
hive_site_path = "/home/hadoop/hive-2.3.9/conf/hive-site.xml"
}
}
transform {
sql {
source_table_name="Table_test"
query = "select xxx,xxx from Table_test"
result_table_name = "Table_test2"
}
}
sink {
Jdbc {
url =
"jdbc:mysql://xxx:3306/xxx?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true&useSSL=false"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
enable_upsert = true
generate_sink_sql = true
password = "xxx"
database = "xxx"
primary_keys = [xxx,xxx,xxx]
table = "xxx"
}
}`
The version I am using is 2.3.1, and the configuration is as shown above,
but the extraction speed is the same as before and has not improved. Is there
something wrong with my configuration?
Approximately 12,000 pieces of data can be extracted per second.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
