Hi Divya,
You don't have the phoenix client-spark JAR in your classpath, which is
required for the phoenix-spark integration to work (as per the
documentation).
As well, you aren't using the vanilla Apache project that this mailing list
supports, but are using a vendor packaged platform (Hortonworks). Since
they maintain their own patches and forks to the upstream Apache versions,
in general you should opt for filing support tickets with them first. In
this particular case, HDP 2.3.4 doesn't actually provide the necessary
phoenix client-spark JAR by default, so your options are limited here.
Again, I recommend filing a support ticket with Hortonworks.
Regards,
Josh
On Sat, Apr 9, 2016 at 9:11 AM, Divya Gehlot <divya.htco...@gmail.com>
wrote:
> Hi,
> The code which I using to connect to Phoenix for writing
> def writeToTable(df: DataFrame,dbtable: String) = {
> val phx_properties = collection.immutable.Map[String, String](
> "zkUrl" -> "localhost:2181:/hbase-unsecure",
> "table" -> dbtable)
>
> df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable)
> }
>
> While Submitting Spark job
> * spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
> --jars
> /usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
> --driver-class-path
> /usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
> /TestDivya/Spark/WriteToPheonix.scala*
>
>
> Getting the below error :
>
> org.apache.spark.sql.AnalysisException: org.apache.phoenix.spark.DefaultSource
> does not allow user-specified schemas.;
>
> Am I on the right track or missing any properties ?
>
> Because of this I am unable to proceed with Phoenix and have to find
> alternate options.
> Would really appreciate the help
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Divya Gehlot <divya.htco...@gmail.com>
> Date: 8 April 2016 at 19:54
> Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
> To: Josh Mahonin <jmaho...@gmail.com>
>
>
> Hi Josh,
> I am doing in the same manner as mentioned in Phoenix Spark manner.
> Using the latest version of HDP 2.3.4 .
> In case of version mismatch/lack of spark Phoenix support it's should have
> thrown the error at read also.
> Which is working fine as expected .
> Will surely pass on the code snippets once I log on to my System.
> In the mean while I would like to know the zkURL parameter.If I build it
> with HbaseConfiguration and passing zk quorom ,znode and port .
> It throws error for example localhost :2181/hbase-unsecure
> This localhost gets replaced by all the quorom
> Like quorum1,quorum2:2181/hbase-unsecure
>
> I am just providing the IP address of my HBase master.
>
> I feel like I am not on right track so asked for the help .
> How to connect to Phoenix through Spark on hadoop cluster .
> Thanks for the help.
> Cheers,
> Divya
> On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jmaho...@gmail.com> wrote:
>
>> Hi Divya,
>>
>> That's strange. Are you able to post a snippet of your code to look at?
>> And are you sure that you're saving the dataframes as per the docs (
>> https://phoenix.apache.org/phoenix_spark.html)?
>>
>> Depending on your HDP version, it may or may not actually have
>> phoenix-spark support. Double-check that your Spark configuration is setup
>> with the right worker/driver classpath settings. and that the phoenix JARs
>> contain the necessary phoenix-spark classes
>> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
>> following up with Hortonworks.
>>
>> Josh
>>
>>
>>
>> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <divya.htco...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I hava a Hortonworks Hadoop cluster having below Configurations :
>>> Spark 1.5.2
>>> HBASE 1.1.x
>>> Phoenix 4.4
>>>
>>> I am able to connect to Phoenix through JDBC connection and able to read
>>> the Phoenix tables .
>>> But while writing the data back to Phoenix table
>>> I am getting below error :
>>>
>>> org.apache.spark.sql.AnalysisException:
>>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>>> schemas.;
>>>
>>> Can any body help in resolving the above errors or any other solution of
>>> saving Spark Dataframes to Phoenix.
>>>
>>> Would really appareciate the help.
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>
