Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15292#discussion_r81665271
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
---
@@ -46,17 +45,18 @@ object JDBCRDD extends Logging {
* Takes a (schema, table) specification and returns the table's Catalyst
* schema.
*
- * @param url - The JDBC url to fetch information from.
- * @param table - The table name of the desired table. This may also be
a
- * SQL query wrapped in parentheses.
+ * @param options - JDBC options that contains url, table and other
information.
*
* @return A StructType giving the table's Catalyst schema.
* @throws SQLException if the table specification is garbage.
* @throws SQLException if the table contains an unsupported type.
*/
- def resolveTable(url: String, table: String, properties: Properties):
StructType = {
+ def resolveTable(options: JDBCOptions): StructType = {
+ val url = options.url
+ val table = options.table
+ val properties = options.asProperties
--- End diff --
Hm, @gatorsmile , first of all, thank you so much for looking into this
close and I now understood that the names are potentially the same. However, I
wonder then if we should exclude all the options apparently handled within
Spark as I guess then they are potentially the same keys for some vendors.
So, I guess you meant we might have to be able to give `url`/`dbtable`
differently as parameters and as properties because they might be conflicted. I
think this implies we should differentiate both options and properties. In this
case, either we should disable the APIs, `read.format("jdbc").option` and
`write.format("jdbc").option` or keep and manage the list of potentially
conflitable key list within Spark.
I am wondering how many cases there would be, in particular, when we should
explicitly set the different values for the same keys as parameters and as
properties and it's worth managing a list or disabling both APIs. (In that case
both take the same format of value, I am explicitly setting a higher precedence
for `url` and `dbtable` given as arguments not in properties.)
I hope I am not making my points wrong or am not nitpicking but this was
what I thought.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
