[
https://issues.apache.org/jira/browse/PHOENIX-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736183#comment-16736183
]
ASF GitHub Bot commented on PHOENIX-5059:
-----------------------------------------
Github user twdsilva commented on the issue:
https://github.com/apache/phoenix/pull/423
@jmahonin Thanks for the review. The Datasource V2 API is marked evolving
and the API changed from Spark 2.3 to 2.4 and it will change again in Spark
3.0. The DataSource v2 connector in this patch will only work with Spark 2.4.
I deprecated but did not remove the current connector that uses the V1 API,
so users should still be able to use the current connector.
There is a JIRA to move all the connectors to their own repo (see
[PHOENIX-5062](https://issues.apache.org/jira/browse/PHOENIX-5062)) which will
allow us to release the connectors independently of phoenix core. I was
planning on working on that after this PR goes in.
For 2) are you referring to ```PhoenixRecordWritable```? That class is
deprecated and only used in the old connector.
Spark has code to convert from/to a ResultSet to/from InternalRow which I
used (see ```SparkJdbcUtil```). The setters methods are accessible, but there
is a bug in handling ```ByteType```, so I had to copy the code. This will be
fixed in [SPARK-26499](https://issues.apache.org/jira/browse/SPARK-26499) after
which we can switch to just using Spark's
```JdbcUtils.resultSetToSparkInternalRows```
The setters methods are not accessible, so I had to create a copy of them
(see ```makeSetter``` in ```SparkJdbcUtil```).
> Use the Datasource v2 api in the spark connector
> ------------------------------------------------
>
> Key: PHOENIX-5059
> URL: https://issues.apache.org/jira/browse/PHOENIX-5059
> Project: Phoenix
> Issue Type: New Feature
> Reporter: Thomas D'Silva
> Assignee: Thomas D'Silva
> Priority: Major
> Attachments: PHOENIX-5059-wip.patch
>
>
> Switching to the v2 api will enable pushing downing limits/aggregations and
> more in the future.
> https://issues.apache.org/jira/browse/SPARK-15689
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
