Reinier Kip created HIVE-18931:

             Summary: Race condition when ensuring a partition exists often 
causes AlreadyExistsException for the partition
                 Key: HIVE-18931
             Project: Hive
          Issue Type: Bug
          Components: Clients, HCatalog
    Affects Versions: 1.2.1
            Reporter: Reinier Kip

Hiya! I'm using Apache Beam's HCatalogIO to store data in Hive. As part of 
HCatOutputFormatWriter#commit(), partitions are registered in 
FileOutputCommitterContainer#registerPartitions(). Here, it checks for the 
existence of the partitions, and, if so needed, creates them. For parallel 
processes, this fails a lot of the time, because, as far as I understand, 
another process creates the partition in the meantime. This causes an 
AlreadyExistsException to be created for the partition.

Relevant logic: 
 Logic is also present on master.

Am I missing something, and if not, is it an acceptable solution to swallow the 
AlreadyExistsException when adding the partitions?

This message was sent by Atlassian JIRA

Reply via email to