By my reading, Hive transactions are fairly primitive (limited form of snapshot isolation) and relatively easy to support.
- for most file formats, they only support insert on unsorted files. What that really means is that new files appear occasionally. For extra points, you can actually look at the transaction manager to see if the files actually should be looked at, but that is a nuance. - for ORC, inserts, updates, and deletes appear as appended elements in a file until it is garbage collected. To determine which of these to apply, reference to the Hive transaction manager is required to get the high watermark as well as a list of pending and aborted transactions to ignore. This should be something that is encapsulated in the ORC scanner. The real point here is that because Hive transactions don't actually mark updated rows and columns, they get away without a lock manager. That also means that you have potential for conflicting writes. At least, that is the way I read it. On Wed, Apr 3, 2019 at 9:25 AM Paul Rogers <[email protected]> wrote: > Hi All, > > Note that Hive 3 has introduced Hive ACID: an innovative way to handle > transactional data on a traditional big data warehouse. Some distros appear > to be talking about enabling ACID by default for all Hive-managed tables. > In order for Drill to continue to work with such tables, Drill may have to > support the Hive ACID protocol. > > Thanks, > - Paul > > > > On Wednesday, April 3, 2019, 1:03:47 AM PDT, Arina Yelchiyeva < > [email protected]> wrote: > > Looks like we don’t have much of a choice if we want to support Hadoop 3. > > Kind regards, > Arina > > > On Apr 2, 2019, at 7:40 PM, Vitalii Diravka <[email protected]> wrote: > > > > Hi devs! > > > > I am working on the update of Hadoop libs to the 3.2.0 version [1]. > > I found the issue in *hadoop-common* related to several loggers in the > > project [2], [3]. > > So to update the version of hadoop libs in Drill it is necessary to > remove > > *commons-logging* from banned dependencies [4]. > > After doing it I didn't find conflicts between two logger libs in Drill. > > > > Is this solution acceptable? > > It can be temporary until [3] is fixed. > > > > > > > > [1] https://issues.apache.org/jira/browse/DRILL-6540 > > [2] > > > https://issues.apache.org/jira/browse/DRILL-6540?focusedCommentId=16606306&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16606306 > > [3] https://issues.apache.org/jira/browse/HADOOP-15749 > > [4] https://github.com/apache/drill/blob/master/pom.xml#L522 > > > > > > Kind regards > > Vitalii
