Hi Jingsong, Thanks for your comments.
> We should document what is based on FLIP-314. I have updated the operations supported by FLIP-314 in the future work > Is the current Source interface sufficient for your functionality? In our design the current Source interface fulfills our requirements. As described in PIP-5, `AlignedEnumerator` will send checkpoint events to `AlignedSourceReader`, which will align the checkpoint and snapshot, and then send split the next operator. More detailed information can be provided by @liming > Can we currently achieve the ability to flush all data in a snapshot before snapshot? Can you provide a more detailed description of this? Do you mean there may be too much data for a snapshot if the source aligns the checkpoint and snapshot and causes the snapshot to be too large to flush? Best, Shammon FY On Mon, Jun 12, 2023 at 4:30 PM Jingsong Li <[email protected]> wrote: > System Database looks very good~ But perhaps there are some design > details here? What API should we use? Paimon Java API? And we should > commit every operation? > > Best, > Jingsong > > On Mon, Jun 12, 2023 at 4:27 PM Jingsong Li <[email protected]> > wrote: > > > > Thanks Shammon, > > > > The overall design looks good to me! > > > > ## Plan For The Future > > > > We should document what is based on FLIP-314. > > > > ## AlignedEnumerator and AlignedSourceReader > > > > Is the current Source interface sufficient for your functionality? > > > > Can we currently achieve the ability to flush all data in a snapshot > > before snapshot? > > > > Best, > > Jingsong > > > > On Mon, Jun 5, 2023 at 7:57 PM Shammon FY <[email protected]> wrote: > > > > > > Hi Kelu, > > > > > > Thanks for your feedback. In the first stage, we do not want to > introduce a > > > server, but instead store information directly in the Paimon table when > > > creating and running Flink jobs. A server will be considered when we > > > encounter more requirements in the future and need a resident service > > > management. > > > > > > Best, > > > Shammon FY > > > > > > On Fri, Jun 2, 2023 at 5:55 PM Kelu Tao <[email protected]> wrote: > > > > > > > +1 > > > > > > > > cool job ~ > > > > > > > > For this PIP, do we need to introduce a new server for the > information > > > > serving? > > > > > > > > On 2023/05/31 02:28:21 Shammon FY wrote: > > > > > Hi devs, > > > > > > > > > > We would like to start a discussion about PIP-5: Paimon Table And > Data > > > > > Lineage For Flink[1]. > > > > > > > > > > As a streaming lake, users can use Paimon integrated with Flink to > > > > complete > > > > > the entire ETL processing. In this process, users need to manage > batch & > > > > > streaming jobs and data streams, including batch & streaming data > > > > > validation, job debug, and data revision. To support the above > ability, > > > > we > > > > > introduce table and data lineage for Flink & Paimon. Users can > > > > conveniently > > > > > manage the entire ETL processing based on lineage information. > > > > > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-5%3A+Paimon+Table+And+Data+Lineage+For+Flink > > > > > > > > > > > > > > > Best, > > > > > Shammon FY > > > > > > > > > >
