[
https://issues.apache.org/jira/browse/PHOENIX-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17873272#comment-17873272
]
ASF GitHub Bot commented on PHOENIX-7377:
-----------------------------------------
rejeb commented on PR #134:
URL:
https://github.com/apache/phoenix-connectors/pull/134#issuecomment-2286761341
I agree with you, the patch is to fix the issue but also to document the new
behavior and add more flexibility in using the spark-phoenix connector.
Please let me now when it is ok for the review. I'll squash the commits in
order to have one commit to merge.
Question, is there an opened issue to remove code duplication and deprecated
classes in spark modules ?
I can help doing it.
> phoenix5-spark dataframe issue with schema inference
> ----------------------------------------------------
>
> Key: PHOENIX-7377
> URL: https://issues.apache.org/jira/browse/PHOENIX-7377
> Project: Phoenix
> Issue Type: Bug
> Reporter: rejeb ben rejeb
> Priority: Major
>
> The fix of the PHOENIX-4981 introduced a bracking change in the way the
> schema was inferred.
> In previous versions of the connector, for non default column family ,
> columns mapped to "columnName" in DataFrame. Now, they are mapped to
> "columnFamily.columnName".
> There are no unit tests that cover this case, all tests uses tables with
> default column family "0".
> The change is made is this [pull
> request|https://github.com/apache/phoenix/pull/402] (the project was moved to
> another git repo since):
> * In previous version code uses `ColumnInfo.getDisplayName` to define the
> name of the column in the DF.
> * The new class SparkSchemaUtil the method used is
> `ColumnInfo.getColumnName` which returns the columnName as
> `columnFamilyName.columnName`.
> The pull request is related to this ticket PHOENIX-4981 the change is not
> documented.
> This change breaks jobs reading from tables having a non default column
> family.
> The saprk3 connector have the same issue since code has been duplicated from
> spark2 module to spark3 module.
> Since V1 api has been modified to use same method to resolve schema it has
> the same behavior and it should not bcause they are now a deprecated classes
> and should not contain a braking change.
>
> *Resolution proposal:*
> The best way to fix the issue is to add a property to have both options for
> schema non default column family column name mapping.
> The issue is in spark connector and it's resolution will not have a side
> effect on other phoenix-connectors like phoenix5-hive for example.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
