Hi Jingsong, Big fan of this idea. We faced the same problem and resolved by adding a distributed lock. It would be nice to have this feature in JobMaster, which can replace the lock.
Best, Shimin JingsongLee <lzljs3620...@aliyun.com.invalid> 于2019年9月6日周五 下午12:20写道： > Hi devs: > > I try to implement streaming file sink for table[1] like StreamingFileSink. > If the underlying is a HiveFormat, or a format that updates visibility > through a metaStore, I have to update the metaStore in the > notifyCheckpointComplete, but this operation occurs on the task side, > which will lead to distributed access to the metaStore, which will > lead to bottleneck. > > So I'm curious if we can support notifyOnMaster for > notifyCheckpointComplete like FinalizeOnMaster. > > What do you think? > > [1] > https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing > > Best, > Jingsong Lee
