Balaji Balasubramaniam created SPARK-37245:
----------------------------------------------
Summary: SAP HANA : Truncate doesn't truncate the table but rather
drops the table
Key: SPARK-37245
URL: https://issues.apache.org/jira/browse/SPARK-37245
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.1.0
Reporter: Balaji Balasubramaniam
Hello,
I currently use AWS Glue 3.0 version. Python version is 3 and the Spark version
is 3.1 and I'm using the latest SAP HANA JDBC driver (ngdbc.jar). I can able to
use this jar to read the data from SAP HANA but when I'm trying to write the
data to SAP HANA, I notice that it is trying to drop the table, rather than
truncating the table. Upon talking to AWS support, they mentioned that HANA
JDBC dialect is not support, thus it is not honoring the truncate option, is
this true?
This is the option that I use:
df_lake.write.format("jdbc").option("url", edw_jdbc_url).option("driver",
"com.sap.db.jdbc.Driver").option("dbtable", edw_jdbc_db_table).option("user",
edw_jdbc_userid).option("password", edw_jdbc_password).option("truncate",
"true").mode("append").save()
This is the stack trace:
Stacktrace : Traceback (most recent call last):
File "/tmp/ejgallo-lake-sds-wsmis-account-segmentation-datalake-to-edw", line
123, in <module>
df_lake.write.jdbc(url=edw_jdbc_url, table=edw_jdbc_db_table,
mode='overwrite', properties=properties)
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py",
line 982, in jdbc
self.mode(mode)._jwrite.jdbc(url, table, jprop)
File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py",
line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line
63, in deco
return f(*a, **kw)
File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py",
line 328, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o165.jdbc.
: com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [258]:
insufficient privilege: Detailed info for this error can be found with guid
'09F34ABDA6F80D4DA539791F19F918A9'
at
com.sap.db.jdbc.exceptions.SQLExceptionSapDB._newInstance(SQLExceptionSapDB.java:191)
at
com.sap.db.jdbc.exceptions.SQLExceptionSapDB.newInstance(SQLExceptionSapDB.java:42)
at
com.sap.db.jdbc.packet.HReplyPacket._buildExceptionChain(HReplyPacket.java:976)
at
com.sap.db.jdbc.packet.HReplyPacket.getSQLExceptionChain(HReplyPacket.java:157)
at
com.sap.db.jdbc.packet.HPartInfo.getSQLExceptionChain(HPartInfo.java:39)
at com.sap.db.jdbc.ConnectionSapDB._receive(ConnectionSapDB.java:3476)
at com.sap.db.jdbc.ConnectionSapDB.exchange(ConnectionSapDB.java:1568)
at
com.sap.db.jdbc.StatementSapDB._executeDirect(StatementSapDB.java:1435)
at com.sap.db.jdbc.StatementSapDB._execute(StatementSapDB.java:1414)
at com.sap.db.jdbc.StatementSapDB._execute(StatementSapDB.java:1399)
at
com.sap.db.jdbc.StatementSapDB._executeUpdate(StatementSapDB.java:1387)
at com.sap.db.jdbc.StatementSapDB.executeUpdate(StatementSapDB.java:175)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.dropTable(JdbcUtils.scala:94)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:61)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at
org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:515)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
endtime_pst : 2021-09-08 00:14:00
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
