ChinmaySKulkarni commented on issue #2: PHOENIX-5197: Use extraOptions to set the configuration for Spark Workers URL: https://github.com/apache/phoenix-connectors/pull/2#issuecomment-479690145 A couple of problems that we will run into: 1. [`DataSourceOptions`](https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceOptions.java#L88-L93) converts all keys in the options map to lowercase. When creating a reader or writer, we create a `DataSourceOptions` object which is passed in all options. See [createReader](https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala#L150) and [createWriter](https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala#L162). This is a problem for passing in a lot of Phoenix configurations like `phoenix.query.timeoutMs` which are case-sensitive. -> We can potentially get past this issue by serializing all configs against a single key in the options map and passing that to the workers. 2. I can look into setting some small values for upsert batch size or query timeouts from the driver to see if the workers honor these values. FYI @twdsilva
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
