Hi Niranda, Thank you for your fast response. The reason I tried to drop table in spark is because I saw spark jira related to it [1]. Anyway thanks for the insight on spark.
[1] - https://issues.apache.org/jira/browse/SPARK-5264 On Thu, Dec 3, 2015 at 7:22 PM, Niranda Perera <[email protected]> wrote: > Hi Amani, Please find my comments inline. > > On Thu, Dec 3, 2015 at 6:29 PM, Amani Soysa <[email protected]> wrote: > >> Hi Niranda/Anjana, >> >> I wanted to change the schema of an existing temporary table in spark, >> however, when I change the schema when defining the table it did not work >> because the table already existed. >> > > this is a known bug in DAS 3.0.0 and it will be resolved in the 3.0.1 > patch release [1]. This happens because currently, the schema of a table is > merged with the existing schema. With the [1] fix, you would get an option > to merge/ replace the schema of a table. > > [1] https://wso2.org/jira/browse/DAS-314 > > >> >> So I tried to drop according to spark manual >> >> "DROP [TEMPORARY] TABLE [IF EXISTS] tbl_name" >> > > I guess you are referring to the MySQL syntax here. In Spark SQL drop > table queries are not supported, because in the spark runtime, a temporary > table is only a mapping to a physical table available in the datasource. > Spark will only pull/ push data to/ from it and it would not delete data. > > >> >> and still it did not work. Finally I had to drop all the H2 tables to >> resolve this issue. >> >> Is there a best practice when changing the schema of spark tables? >> > > Unfortunately, until the DAS-314 fix is released, you would have to create > a new table with the changed schema. > > >> >> >> Regards, >> Amani >> -- >> Amani Soysa >> Associate Technical Lead >> Mobile: +94772325528 >> WSO2, Inc. | http://wso2.com/ >> Lean . Enterprise . Middleware >> > > > Hope I answered your question. > > Rgds > > -- > *Niranda Perera* > Software Engineer, WSO2 Inc. > Mobile: +94-71-554-8430 > Twitter: @n1r44 <https://twitter.com/N1R44> > https://pythagoreanscript.wordpress.com/ > -- Amani Soysa Associate Technical Lead Mobile: +94772325528 WSO2, Inc. | http://wso2.com/ Lean . Enterprise . Middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
