> On March 31, 2015, 4:36 p.m., Srikanth Sundarrajan wrote: > > common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java, > > line 191 > > <https://reviews.apache.org/r/32688/diff/1/?file=911160#file911160line191> > > > > Understand that this will handle the cases of re-run where data is > > re-added. In that scenario, if drop partitions are complete, but add > > partition where to fail mid way, would it leave the table in a bad state ? > > This might be a large gap, if this isn't atomic. If this is indeed a valid > > issue, we need to address this. But given the largish nature of the fix, we > > can file another JIRA and handle this separately.
Hive 0.14 supports transactional insert, delete and update. Meaning, inserts, updates and deletes can happen in batches - https://issues.apache.org/jira/browse/HIVE-5317. This we should be leveraging at the earliest. They also have a plan to come up with support for user demarcated transaction boundaries that we can leverage later - https://issues.apache.org/jira/browse/HIVE-9675 Have filed FALCON-1130 for the same. - Pallavi ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32688/#review78368 ----------------------------------------------------------- On March 31, 2015, 11:35 a.m., Srikanth Sundarrajan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/32688/ > ----------------------------------------------------------- > > (Updated March 31, 2015, 11:35 a.m.) > > > Review request for Falcon. > > > Bugs: FALCON-1091 > https://issues.apache.org/jira/browse/FALCON-1091 > > > Repository: falcon-git > > > Description > ------- > > FALCON-1091 Monitoring plugin code > > > Diffs > ----- > > common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java > 9abdc93 > common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java > PRE-CREATION > common/src/main/java/org/apache/falcon/catalog/HiveCatalogService.java > 25a4a46 > common/src/main/java/org/apache/falcon/entity/CatalogStorage.java 59f558b > common/src/main/java/org/apache/falcon/entity/FeedHelper.java ca31f95 > common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java > 1ba7b9d > common/src/main/java/org/apache/falcon/entity/common/FeedDataPath.java > 6ededbb > common/src/main/java/org/apache/falcon/expression/ExpressionHelper.java > 33ec59c > common/src/main/java/org/apache/falcon/util/FalconRadixUtils.java 4bf6e00 > > common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java > 8d69b9a > common/src/main/resources/startup.properties 99dab59 > common/src/test/java/org/apache/falcon/entity/FeedDataPathTest.java c405556 > common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java f6994fc > common/src/test/java/org/apache/falcon/entity/FileSystemStorageTest.java > 1667161 > > hadoop-dependencies/src/versioned-src/v2/java/org/apache/hadoop/mapred/ClassicClientProtocolProvider.java > 2167375 > oozie/src/main/java/org/apache/falcon/logging/LogProvider.java 2e5dffb > > oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java > 7a87919 > > oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java > 462e26b > > oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java > 545beb1 > prism/pom.xml 4a3054a > retention/src/test/java/org/apache/falcon/retention/FeedEvictorTest.java > 970d381 > test-tools/hadoop-webapp/src/main/resources/mapred-site.xml cf297de > test-tools/hadoop-webapp/src/main/resources/yarn-site.xml 658752b > > webapp/src/test/java/org/apache/falcon/catalog/CatalogPartitionHandlerIT.java > PRE-CREATION > webapp/src/test/java/org/apache/falcon/catalog/HiveCatalogServiceIT.java > 71616e9 > > webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java > 6982b65 > webapp/src/test/java/org/apache/falcon/util/HiveTestUtils.java 19274b9 > webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java e67fe2a > webapp/src/test/resources/cluster-template.xml 16b7c8c > webapp/src/test/resources/feed-template1.xml 456f7ce > webapp/src/test/resources/feed-template2.xml d4901fa > > Diff: https://reviews.apache.org/r/32688/diff/ > > > Testing > ------- > > > Thanks, > > Srikanth Sundarrajan > >
