This is for Hive not for Spark, there updates from different as updates on rdbms. What they actually do they add it and the end of your file and then it uses a compaction process that only keeps the last record.
Take a look at : https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions <https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions> "Basic Design HDFS does not support in-place changes to files. It also does not offer read consistency in the face of writers appending to files being read by a user. In order to provide these features on top of HDFS we have followed the standard approach used in other data warehousing tools. Data for the table or partition is stored in a set of base files. New records, updates, and deletes are stored in delta files. A new set of delta files is created for each transaction (or in the case of streaming agents such as Flume or Storm, each batch of transactions) that alters a table or partition. At read time the reader merges the base and delta files, applying any updates and deletes as it reads. " Jorge Machado www.jmachado.me > On 29/03/2016, at 10:27, Sage Meng <lkke...@gmail.com> wrote: > > Hi, > > according to > https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-NewConfigurationParametersforTransactions > > <https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-NewConfigurationParametersforTransactions> > it seems that insert/delete/update can be supported, but I haven't try it. > > 2016-03-29 16:23 GMT+08:00 Jorge Machado <jom...@me.com > <mailto:jom...@me.com>>: > Hi, > > you should know that “Spark” is not a relation database. So updates on data > as you are used to in RDMS are not possible. > > Jorge Machado > www.jmachado.me <http://www.jmachado.me/> > > > > > >> On 29/03/2016, at 10:21, Sage Meng <lkke...@gmail.com >> <mailto:lkke...@gmail.com>> wrote: >> >> thanks, I found that I can use hive's jdbc driver to connect to spark sql. >> >> I am curious whether simba's jdbc/odbc drivers to spark sql can support all >> standard sql statements, since I haven't tried third-party's jdbc/odbc >> driver and it seems that hive's jdbc driver can't support >> insert/delete/update sql well. >> >> 2016-03-29 15:47 GMT+08:00 alexpw <al...@simba.com <mailto:al...@simba.com>>: >> sage wrote >> > Hi all, >> > Does SparkSql has official jdbc/odbc driver? I only saw third-party's >> > jdbc/odbc driver. >> >> Hi Sage, >> >> Databricks licenses ODBC driver from Simba Technologies. Here's the link to >> announcement: >> https://databricks.com/blog/2014/04/30/databricks-partners-with-simba-to-deliver-shark-odbc-driver.html >> >> <https://databricks.com/blog/2014/04/30/databricks-partners-with-simba-to-deliver-shark-odbc-driver.html> >> >> The driver is available from Databricks website for free: >> https://databricks.com/spark/odbc-driver-download >> <https://databricks.com/spark/odbc-driver-download> but is licensed for use >> only within Databricks Cloud: >> >> "3. Scope of use: You can use this driver only to connect a third party >> application (including Microsoft Excel or Tableau) to Spark SQL within >> Databricks Cloud, using the ODBC protocol." >> >> You can read the Terms of Use here: >> https://databricks.wufoo.com/embed/p1xbn36500t2zce/#fo7li1501 >> <https://databricks.wufoo.com/embed/p1xbn36500t2zce/#fo7li1501> >> >> If you plan to use ODBC driver with Spark on any other platform, you can >> purchase license from Simba Technologies. Simba also have JDBC driver for >> Spark built use the same SQL Engine as the ODBC. Both drivers are available >> for free evaluation and purchase here: >> http://www.simba.com/drivers/spark-jdbc-odbc/ >> <http://www.simba.com/drivers/spark-jdbc-odbc/> >> >> Thanks, >> >> Alex Popov >> Marketing Manager, Simba Technologies Inc. >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Does-SparkSql-has-official-jdbc-odbc-driver-tp26591p26620.html >> >> <http://apache-spark-user-list.1001560.n3.nabble.com/Does-SparkSql-has-official-jdbc-odbc-driver-tp26591p26620.html> >> Sent from the Apache Spark User List mailing list archive at Nabble.com >> <http://nabble.com/>. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> <mailto:user-unsubscr...@spark.apache.org> >> For additional commands, e-mail: user-h...@spark.apache.org >> <mailto:user-h...@spark.apache.org> >> >> > >
