Ostap Gonchar created SPARK-21338:
-------------------------------------
Summary: AggregatedDialect doesn't override
isCascadingTruncateTable() method
Key: SPARK-21338
URL: https://issues.apache.org/jira/browse/SPARK-21338
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.1.1, 2.1.0
Reporter: Ostap Gonchar
org.apache.spark.sql.jdbc.JdbcDialect's method:
def isCascadingTruncateTable(): Option[Boolean] = None
is not overriden in org.apache.spark.sql.jdbc.AggregatedDialect class.
Because of this issue - *+when you add more than one dialect Spark doesn't
truncate table because isCascadingTruncateTable always returns default None for
Aggregated Dialect+*.
Can be tracked when you write dataset in jdbc mode with SaveMode.Overwrite and
extra JdbcDialect added via JdbcDialects.registerDialect(...); method.
Then inside of
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider on line 65
in createRelation method it will never get to line 67 even if isTruncate
returns true.
It will only get inside if you have one JdbcDialect and AggregatedDialect is
not used.
Fix would be to overrite this method in AggregatedDialect and also to reduce
value (I guess with and clause same as in overriden getHandle method) e.g.
override def isCascadingTruncateTable(): Option[Boolean]=
Some(dialects.map(_.isCascadingTruncateTable()).reduce(_ && _))
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
