Hi, Jira created: https://issues.apache.org/jira/browse/SPARK-6675
Thank you. On Wed, Apr 1, 2015 at 7:50 PM, Michael Armbrust <mich...@databricks.com> wrote: > Can you open a JIRA please? > > On Wed, Apr 1, 2015 at 9:38 AM, Hao Ren <inv...@gmail.com> wrote: > >> Hi, >> >> I find HiveContext.setConf does not work correctly. Here are some code >> snippets showing the problem: >> >> snippet 1: >> >> ---------------------------------------------------------------------------------------------------------------- >> import org.apache.spark.sql.hive.HiveContext >> import org.apache.spark.{SparkConf, SparkContext} >> >> object Main extends App { >> >> val conf = new SparkConf() >> .setAppName("context-test") >> .setMaster("local[8]") >> val sc = new SparkContext(conf) >> val hc = new HiveContext(sc) >> >> *hc.setConf("spark.sql.shuffle.partitions", "10")* >> * hc.setConf("hive.metastore.warehouse.dir", >> "/home/spark/hive/warehouse_test")* >> hc.getAllConfs filter(_._1.contains("warehouse.dir")) foreach println >> hc.getAllConfs filter(_._1.contains("shuffle.partitions")) foreach >> println >> } >> >> ---------------------------------------------------------------------------------------------------------------- >> >> *Results:* >> (hive.metastore.warehouse.dir,/home/spark/hive/warehouse_test) >> (spark.sql.shuffle.partitions,10) >> >> snippet 2: >> >> ---------------------------------------------------------------------------------------------------------------- >> ... >> *hc.setConf("hive.metastore.warehouse.dir", >> "/home/spark/hive/warehouse_test")* >> * hc.setConf("spark.sql.shuffle.partitions", "10")* >> hc.getAllConfs filter(_._1.contains("warehouse.dir")) foreach println >> hc.getAllConfs filter(_._1.contains("shuffle.partitions")) foreach >> println >> ... >> >> ---------------------------------------------------------------------------------------------------------------- >> >> *Results:* >> (hive.metastore.warehouse.dir,/user/hive/warehouse) >> (spark.sql.shuffle.partitions,10) >> >> *You can see that I just permuted the two setConf call, then that leads >> to two different Hive configuration.* >> *It seems that HiveContext can not set a new value on >> "hive.metastore.warehouse.dir" key in one or the first "setConf" call.* >> *You need another "setConf" call before changing >> "hive.metastore.warehouse.dir". For example, set >> "hive.metastore.warehouse.dir" twice and the snippet 1* >> >> snippet 3: >> >> ---------------------------------------------------------------------------------------------------------------- >> ... >> * hc.setConf("hive.metastore.warehouse.dir", >> "/home/spark/hive/warehouse_test")* >> * hc.setConf("hive.metastore.warehouse.dir", >> "/home/spark/hive/warehouse_test")* >> hc.getAllConfs filter(_._1.contains("warehouse.dir")) foreach println >> ... >> >> ---------------------------------------------------------------------------------------------------------------- >> >> *Results:* >> (hive.metastore.warehouse.dir,/home/spark/hive/warehouse_test) >> >> >> *You can reproduce this if you move to the latest branch-1.3 >> (1.3.1-snapshot, htag = 7d029cb1eb6f1df1bce1a3f5784fb7ce2f981a33)* >> >> *I have also tested the released 1.3.0 (htag = >> 4aaf48d46d13129f0f9bdafd771dd80fe568a7dc). It has the same problem.* >> >> *Please tell me if I am missing something. Any help is highly >> appreciated.* >> >> Hao >> >> -- >> Hao Ren >> >> {Data, Software} Engineer @ ClaraVista >> >> Paris, France >> > > -- Hao Ren {Data, Software} Engineer @ ClaraVista Paris, France