[
https://issues.apache.org/jira/browse/SPARK-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-20394.
------------------------------------
Resolution: Workaround
Great. I doubt we'll fix this in 1.6 at this point, and I believe this is fixed
in 2.x, so let's close this.
> Replication factor value Not changing properly
> ----------------------------------------------
>
> Key: SPARK-20394
> URL: https://issues.apache.org/jira/browse/SPARK-20394
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Submit
> Affects Versions: 1.6.0
> Reporter: Kannan Subramanian
>
> To save SparkSQL dataframe to a persistent hive table using the below steps.
> a) RegisterTempTable to the dataframe as a tempTable
> b) create table <table name> (cols....)PartitionedBy(col1, col2) stored as
> parquet
> c) Insert into <table name> partition(col1, col2) select * from tempTable
> I have set dfs.replication is equal to "1" in hiveContext object. But It did
> not work properly. That is replica is 1 for 80 % of the generated parquet
> files on HDFS and default replica 3 is for remaining 20 % of parquet files in
> HDFS. I am not sure why the replica is not reflecting to all the generated
> parquet files. Please let me know if you have any suggestions or solutions
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
