Hi Amani, Please find my comments inline. On Thu, Dec 3, 2015 at 6:29 PM, Amani Soysa <[email protected]> wrote:
> Hi Niranda/Anjana, > > I wanted to change the schema of an existing temporary table in spark, > however, when I change the schema when defining the table it did not work > because the table already existed. > this is a known bug in DAS 3.0.0 and it will be resolved in the 3.0.1 patch release [1]. This happens because currently, the schema of a table is merged with the existing schema. With the [1] fix, you would get an option to merge/ replace the schema of a table. [1] https://wso2.org/jira/browse/DAS-314 > > So I tried to drop according to spark manual > > "DROP [TEMPORARY] TABLE [IF EXISTS] tbl_name" > I guess you are referring to the MySQL syntax here. In Spark SQL drop table queries are not supported, because in the spark runtime, a temporary table is only a mapping to a physical table available in the datasource. Spark will only pull/ push data to/ from it and it would not delete data. > > and still it did not work. Finally I had to drop all the H2 tables to > resolve this issue. > > Is there a best practice when changing the schema of spark tables? > Unfortunately, until the DAS-314 fix is released, you would have to create a new table with the changed schema. > > > Regards, > Amani > -- > Amani Soysa > Associate Technical Lead > Mobile: +94772325528 > WSO2, Inc. | http://wso2.com/ > Lean . Enterprise . Middleware > Hope I answered your question. Rgds -- *Niranda Perera* Software Engineer, WSO2 Inc. Mobile: +94-71-554-8430 Twitter: @n1r44 <https://twitter.com/N1R44> https://pythagoreanscript.wordpress.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
