[
https://issues.apache.org/jira/browse/SPARK-37997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482192#comment-17482192
]
Hyukjin Kwon commented on SPARK-37997:
--------------------------------------
Can we just format it before passing to spark.read?
> Allow query parameters to be passed into spark.read
> ---------------------------------------------------
>
> Key: SPARK-37997
> URL: https://issues.apache.org/jira/browse/SPARK-37997
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: QFW
> Priority: Minor
>
> This ticket is to allow query parameters to be used spark.read.
> While it is possible to inject some parameters into the query using string
> concatenation, this doesn't work for all data types, for example binaries. In
> this example, the parameter rowversion is a binary which needs to be passed
> into the sql query.
> {code:java}
> _select_sql = f'SELECT * FROM dbo.Table WHERE RowVersion > {rowversion}'
> df = spark.read.format("jdbc") \
> .option("url",
> "jdbc:sqlserver://databaseserver.database.windows.net;databaseName=databasename")
> \
> .option("query", _select_sql) \
> .option("username", "sql_username") \
> .option("password", "sql_password") \
> .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
> .load() {code}
> This results in the query looking this this...
> {code:java}
> SELECT * FROM dbo.Address WHERE RowVersion >
> bytearray(b'\x00\x00\x00\x00\x02\xdf=\xf5') {code}
> As far as I know, there is no way to do this currently.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
