[
https://issues.apache.org/jira/browse/PHOENIX-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436825#comment-15436825
]
Xiaopeng Liao commented on PHOENIX-2648:
----------------------------------------
Initially the ":" is used to indicate the dynamic column type, however, the
indexTools unit test shows that it is affected by that, so it is changed to
"<".
And for the dataframe, it uses catalyst type, so to convert them into Phoenix
type is a bit tricky, there is a method for reversed direction, so it is used
as reference ... looking for a better way to do it..
And in spark, if it couldn't find the column when saving DataFrame, it will
create that column as dynamic column, not sure if we want to have a
configuration on that behavior so that we can configure it to throw an
exception saying column not found..
> Phoenix Spark Integration does not allow Dynamic Columns to be mapped
> ---------------------------------------------------------------------
>
> Key: PHOENIX-2648
> URL: https://issues.apache.org/jira/browse/PHOENIX-2648
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.6.0
> Environment: phoenix-spark-4.6.0-HBase-0.98 ,
> spark-1.5.0-bin-hadoop2.4
> Reporter: Suman Datta
> Labels: patch, phoenixTableAsRDD, spark
> Fix For: 4.6.0
>
>
> I am using spark-1.5.0-bin-hadoop2.4 and phoenix-spark-4.6.0-HBase-0.98 to
> load phoenix tables on hbase to Spark RDD. Using the steps in
> https://phoenix.apache.org/phoenix_spark.html, I can successfully map
> standard columns in a table to Phoenix RDD.
> But my table has some important dynamic columns
> (https://phoenix.apache.org/dynamic_columns.html) which are not getting
> mapped to Spark RDD in this process.(using sc.phoenixTableAsRDD)
> This is proving a showstopper for me for using phoenix with spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
