Hi Fabian, Thanks for the great FLIP! One quick question on the probe-side restriction.
The FLIP requires the probe (left) side to be append-only. I understand this is for correctness, since a retraction on the probe side could be joined against a different build-side version than the original insert. But I wonder if this restriction is a bit too strong. Today's lookup join (also a processing-time temporal join) already allows a changelog/CDC stream on its input without this constraint. Could we offer a similar opt-in config for LATERAL SNAPSHOT, so that users who can tolerate the version-mismatch risk are also able to use a CDC stream on the left side? Best regards, Hongshun On Wed, Jun 3, 2026 at 1:37 AM Fabian Hueske <[email protected]> wrote: > Thanks for your valuable feedback Jark and Leonard! > > You are bringing up three of the tricky challenges that the new join needs > to deal with. > > (1) Jark: The build-side flip point is not exact > > This is correct. However, I would argue that a processing-time join does > not have exact guarantees anyway and can only produce roughly time-aligned > results. WM alignment of build and probe-side input should help to keep the > alignment somewhat close. Of course, this does not mean that we shouldn't > try to give as good guarantees as possible. > > The primary mechanism for flipping from LOAD to JOIN phase is the > build-side watermark crossing a configured point in time. Watermarks are > used to track progress and completeness in Flink. Using them as a condition > to switch from LOAD to JOIN phase, means that the build-side received at > least all changes up to that point in time. There might have been changes > with later timestamps as well. These could be buffered on the side to have > a stricter FLIP point, but IMO this additional data should be tolerable > under proc-time semantics. > > If the build-side input becomes stale, the processing idle timeout flip > condition gets applied. The assumption is that the build-side source is > currently exhausted and all data was consumed but the WM didn't progress > far enough to exceed the flip point. In this case, we want to flip and > start the regular JOIN phase. > > For the use case of sources with an exact flip point, users would need to > know the timestamp of the last backlog record (or compute it if they know > roughly how long it takes to scan the backlog if it is computed on-demand). > I agree, this is not very practical. > I can think of two options > a) the source connector emits a special WM when it reaches the end of the > backlog. This would not require changes to the join operator but to the > source connectors. > b) The design of the SNAPSHOT function has the `load_completed_condition` > which is an extension point to add logic to determine the flip point. > > > (2) Jark: Buffering probe-side during LOAD phase > > I think this is very similar to Leonard's point about "LOAD phase > backpressure" and also closely related to Leonard's point about "Flip-point > CPU spike". > > This is indeed a potential problem. If used without care, the probe-side > state might grow very large. Before talking about possible ways to address > this problem, let me explain how I think that the join would be used. > > A very common (maybe the most common?) use case should be to initialize the > build-side input up to time t_b and then start processing the probe-side > input from time t_p (with t_p = t_b, or slightly less than t_b) onwards. WM > alignment would help to roughly align build-side and probe-side inputs > (although not being perfectly aligned like the event-time join) > Initializing the build-side up to t_b and starting consuming the probe-side > from t_p with (t_p << t_b) would mean that the first probe-side records are > joined with much later versions of the build-side. > > The first scenario (t_p = t_b) can be controlled with WM alignment and > scan-offset table hints on the probe-side input. Since the WMs of the > build-side input would be less than the WMs of the probe-side input, the > probe-side input should be throttled until the build-side caught up (which > should be close the the flip point). > Other scenarios (including the t_p << t_b scenario) would benefit from an > idea [1] that is described in the future work section of the FLIP. That > mechanism would also be based on WM alignment and would need some > collaboration from the build-side source operator to indicate completeness > of the backlog. > > Using the InputSelectable interface is an idea that we also looked into (as > Gustavo already pointed out). Unfortunately, it is incompatible with > unaligned checkpoints and there are no other streaming operators that > implement the interface. I haven't looked in depth at the current > limitations, but if some of these would be resolved, it might be possible > to later extend the join operator. It might even work with relaxed > guarantees because we don't need to fully block the input but just throttle > it such that less probe-side data needs to be buffered. > > The idea of limiting the size of the probe-side buffer with a config like > `max-buffer-size` sounds interesting. However, I'm not sure if applying > backpressure would really work because we still need to consume the > build-side to be able to reach the flip point and selective backpressure is > not possible without InputSelectible. > > An earlier draft of the proposal described eager joining (now moved to the > future work section [2]). The idea is that a probe-side record would be > directly joined when a match was present or received during the LOAD phase. > After joining it wouldn't be put into state. This would of course > significantly reduce the state size and solved the issue of CPU spikes > during the transition but come at the cost of hard-to-explain semantics > (the current semantics are rather simple, we collect until the flip point > and join against that). > Also, the idea of eager joining was developed when the join operator was > still restricted to FK-PK joins (single build-side record per join key). > The design generalized this restriction to arbitrary joins which means > there might be more build-side matches for a probe-side record such that > the presence of a single join match (of a possibly earlier version) does > not guarantee completeness anymore. That's why the idea of eager joining > was discarded for now. > > > (3) Leonard: Flip-point CPU spikes > > This is also a very valid concern. I would argue that the primary mechanism > to address this point should be to reduce the amount of buffered probe-side > records (see point (2) above). > > I also thought about your idea to micro-batch the draining. In the design, > the transition join is triggered per-key by event-time timers that also > emit the "current" probe-side WM downstream. If we want to continue using > this mechanism, we would need to schedule multiple timers and probably use > some clever mechanism to gradually advance probe-side WMs while still > consuming records. There could be a separate "TRANSITION" phase during > which we still append to the probe-buffer but use the mechanism for atomic > build-side updates. However, this would significantly complicate the design > affecting the control flow and recovery. Hence, I would first try to > address this issue by reducing the probe-side state. > > If you have better ideas for how to use micro-batching during flip > transition, I'm very open to exploring those. > > Leonard also brought up a very important point about metrics! > I will add a section on operator metrics that will help to understand the > state of the operator. > > Please let me know your thoughts! > > Best, Fabian > > [1] > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=421958523#FLIP579:LATERALSNAPSHOTJoin-ReduceBufferingofProbe-SideviaBuild-SideWatermarkSuppression > [2] > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=421958523#FLIP579:LATERALSNAPSHOTJoin-EagerjoinmodeduringLOADphase > > > Am Mo., 1. Juni 2026 um 16:44 Uhr schrieb Leonard Xu <[email protected]>: > > > Hi Fabian > > > > Thanks for driving this FLIP and your kind patience. The motivation is > > spot-on, and the LOAD→JOIN two-phase design is the right structural fix > for > > the FLINK-19830 initialization problem. Overall direction +1 from my > side. > > > > Besides Jark’s idea about backlog and InputSelectable which may need more > > prerequisites, I’ve two concerns about current proposal: > > > > 1. LOAD phase backpressure. The FLIP assumes "seconds to a few minutes" > > for build-side init, but nothing enforces it. Large build-side tables > > (e.g., 50M rows) + fast probe streams → unbuffered probe-side state > > explosion. Should we add a config like max-buffer-size that applies > > backpressure when exceeded or some metrics about buffer, rather than > > silently piling up records? > > > > 2. Flip-point CPU spike. Joining all buffered probe records × build-side > > state in one shot differs fundamentally from event-time join's > incremental > > watermark-batched emission. In the worst case this could cause a > > TaskManager CPU spike and downstream shock. Worth considering micro-batch > > draining during flip transition? > > > > Looking forward to your thoughts. > > > > Best, > > Leonard > > > > > 2026 6月 1 16:02，Fabian Hueske <[email protected]> 写道： > > > > > > Hi Leonard, > > > > > > Sorry, missed your email and already started the vote. > > > Let me put it on hold for now and continue discussing the proposal. > > > > > > Looking forward to your comments, > > > Fabian > > > > > > Am Mo., 1. Juni 2026 um 09:56 Uhr schrieb Leonard Xu < > [email protected] > > >: > > > > > >> @Fabian Thanks for driving this FLIP, sorry for late reply due to my > > >> personal reason that I shouldn’t miss such an important FLIP. > > >> > > >> I’m reviewing the FLIP and will try to finish it today, could you > kindly > > >> wait one minute to start the vote? > > >> > > >> And sorry for interrupt your plan again. > > >> > > >> Best, > > >> Leonard > > >> > > >>> 2026 6月 1 15:51，Fabian Hueske <[email protected]> 写道： > > >>> > > >>> Thanks everyone for your comments on the FLIP. > > >>> I will start the vote. > > >>> > > >>> Best, Fabian > > >>> > > >>> Am Do., 28. Mai 2026 um 20:13 Uhr schrieb David Anderson < > > >>> [email protected]>: > > >>> > > >>>> Fabian, > > >>>> > > >>>>> So, I don't think that we should buffer unmatched probe-side > records > > >>>> beyond > > >>>> the flip point. > > >>>> > > >>>> Thanks for explaining your reasoning. Makes sense to me. > > >>>> > > >>>> David > > >>>> > > >>>> On Thu, May 28, 2026 at 6:55 PM Fabian Hueske <[email protected]> > > >> wrote: > > >>>> > > >>>>> Hi Xingcan, > > >>>>> > > >>>>> Thanks for your comments on the FLIP! > > >>>>> > > >>>>> The join's behavior when starting from a savepoint is indeed an > > >> important > > >>>>> aspect to consider and the problem of a rapidly advancing dimension > > >>>>> (build-side) table is of course real. > > >>>>> > > >>>>> I would argue that watermark alignment should significantly reduce > > the > > >>>>> impact of this. > > >>>>> If enabled, sources align their consumption based on their current > > >>>>> watermark such that the (presumably much smaller) build-side source > > >> would > > >>>>> be slowed down to the event-time progress of the probe-side. > > >>>>> While watermark alignment is not an "exact" mechanism, the > semantics > > of > > >>>> the > > >>>>> new processing-time join also do not guarantee "exact" results. > > >>>>> At the same time, alignment should ensure that build and probe-side > > are > > >>>>> roughly aligned in event-time (without the strict guarantees that > the > > >>>>> event-time temporal table join provides). > > >>>>> > > >>>>> However, I really like your idea of starting in event-time mode and > > >>>>> flipping to processing-time after the initialization duration > passed. > > >>>>> I'm not sure if it would fully address the problem you described. > As > > >> you > > >>>>> said, users would need to be able to reconfigure the flip-point and > > I'm > > >>>> not > > >>>>> sure if there's a good mechanism for this yet. > > >>>>> But it might have some other properties that would be beneficial, > so > > >> I'll > > >>>>> think about that. > > >>>>> > > >>>>> Best, > > >>>>> Fabian > > >>>>> > > >>>>> > > >>>>> Am Do., 28. Mai 2026 um 18:21 Uhr schrieb Fabian Hueske < > > >>>> [email protected] > > >>>>>> : > > >>>>> > > >>>>>> Thanks for your feedback David! > > >>>>>> > > >>>>>>> One question: If I understand correctly, during the JOIN phase of > > an > > >>>>>> INNER > > >>>>>> join, if the desired build-side record is missing, nothing will be > > >>>>> emitted > > >>>>>> for the unmatched probe-side record. For an INNER join, I can > > imagine > > >>>>>> wanting to buffer unmatched probe-side records, expecting the > build > > >>>> side > > >>>>>> will arrive soon. What's your thinking there? > > >>>>>> > > >>>>>> Your understanding is correct. If a probe-side record arrives > during > > >>>> LOAD > > >>>>>> phase but no matching build-side record is received, > > >>>>>> the probe-side record would be discarded without being joined > during > > >>>> the > > >>>>>> transition from LOAD to JOIN. > > >>>>>> > > >>>>>> I would argue that users that want to prevent this, would need to > > >>>>>> configure a longer initialization time. > > >>>>>> IMO, dropping unmatched probe records is not a "bad" property of > > INNER > > >>>>>> joins but an essential part of their semantics. It might even be > > >>>> desired > > >>>>> by > > >>>>>> some users. > > >>>>>> If we would buffer probe-side records for INNER joins beyond the > > >>>>>> transition point, we: > > >>>>>> * would have different behaviors for INNER and LEFT joins > > >>>>>> * could not start to emit probe-side watermarks as long as there > are > > >>>>> still > > >>>>>> probe-side records buffered (or at least not advance past them > > without > > >>>>>> emitting late data at a later point of time) > > >>>>>> * would either need another config knob to specify when to > "really" > > >>>> clean > > >>>>>> up the probe-side state or keep such unmatched records forever in > > >> state > > >>>>> (we > > >>>>>> could also use state TTL...) > > >>>>>> > > >>>>>> So, I don't think that we should buffer unmatched probe-side > records > > >>>>>> beyond the flip point. > > >>>>>> > > >>>>>> Best, Fabian > > >>>>>> > > >>>>>> Am Do., 28. Mai 2026 um 17:05 Uhr schrieb Xingcan Cui < > > >>>>> [email protected] > > >>>>>>> : > > >>>>>> > > >>>>>>> Hi Fabian, > > >>>>>>> > > >>>>>>> Thanks for this FLIP! The two-phase design is excellent for > > avoiding > > >>>>>>> early-joining bugs while maintaining low-latency processing-time > > >>>>>>> semantics. > > >>>>>>> > > >>>>>>> After thinking more about the proposal, I'd like to point out an > > edge > > >>>>> case > > >>>>>>> related to the initialization phase or recovery after prolonged > > >>>> downtime > > >>>>>>> (for example, when a job has been down for a day). While a > > >>>>> processing-time > > >>>>>>> join works well for live streaming, where results can reasonably > > >>>> depend > > >>>>> on > > >>>>>>> the immediate arrival order of live data, it does not work as > well > > >> for > > >>>>>>> catch-up scenarios. > > >>>>>>> > > >>>>>>> Currently, if a job initializes or restores from a checkpoint > > after a > > >>>>> long > > >>>>>>> downtime, the operator resumes directly in the processing-time > join > > >>>>> phase. > > >>>>>>> During catch-up, however, the natural chronological arrival order > > of > > >>>> the > > >>>>>>> live data is completely lost. As a result, these replayed fact > > >> records > > >>>>> are > > >>>>>>> evaluated against the current machine time and may blindly join > > with > > >>>> the > > >>>>>>> rapidly advancing "current" dimension snapshot, rather than the > > >>>>> historical > > >>>>>>> versions they were originally supposed to match. > > >>>>>>> > > >>>>>>> To handle this edge case, could we consider: > > >>>>>>> > > >>>>>>> 1. changing the first phase into an event-time join phase, and > > >>>>>>> > > >>>>>>> 2. allowing the operator to switch back to the first phase after > a > > >>>>>>> restart? > > >>>>>>> > > >>>>>>> For example, users could configure a timestamp threshold. Before > > the > > >>>>>>> watermark reaches that point, the operator would run as an > > event-time > > >>>>>>> versioned join to safely process the catch-up phase through > > watermark > > >>>>>>> alignment. Once the watermark passes the threshold, the operator > > >> could > > >>>>>>> purge the old multi-version state and seamlessly transition back > to > > >>>> the > > >>>>>>> pure processing-time join phase for live traffic. > > >>>>>>> > > >>>>>>> After a job restart, users could either update the target > timestamp > > >> to > > >>>>>>> reset the operator back into the event-time phase, or leave it > > >>>> unchanged > > >>>>>>> to > > >>>>>>> continue operating in the processing-time phase. > > >>>>>>> > > >>>>>>> I completely understand that this would introduce significant > > >>>> complexity > > >>>>>>> to > > >>>>>>> the operator's state management and lifecycle, so this is only a > > >>>>> tentative > > >>>>>>> proposal to explore whether it might be worth considering for the > > >>>>>>> long-term > > >>>>>>> robustness of the design. > > >>>>>>> > > >>>>>>> Best, > > >>>>>>> > > >>>>>>> Xingcan > > >>>>>>> > > >>>>>>> On Thu, May 28, 2026 at 8:17 AM David Anderson < > > [email protected] > > >>> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> I'm quite enthusiastic about this. I want to thank Fabian for > > >>>> putting > > >>>>>>>> together such a well-crafted FLIP. And I look forward to > updating > > >>>> the > > >>>>>>>> awkward educational content this FLIP will make obsolete. > > >>>>>>>> > > >>>>>>>> To my mind, the syntax expresses the semantics of this join > rather > > >>>>> well. > > >>>>>>>> > > >>>>>>>> Until now, developers using event-time temporal joins sometimes > > >>>>>>> resorted to > > >>>>>>>> doing weird things with watermarks to handle a build side that's > > >>>>> mostly > > >>>>>>>> idle; this lateral snapshot join is clearly better -- not to > > mention > > >>>>> the > > >>>>>>>> added bonus of pre-loading the build table. > > >>>>>>>> > > >>>>>>>> One question: If I understand correctly, during the JOIN phase > of > > an > > >>>>>>> INNER > > >>>>>>>> join, if the desired build-side record is missing, nothing will > be > > >>>>>>> emitted > > >>>>>>>> for the unmatched probe-side record. For an INNER join, I can > > >>>> imagine > > >>>>>>>> wanting to buffer unmatched probe-side records, expecting the > > build > > >>>>> side > > >>>>>>>> will arrive soon. What's your thinking there? > > >>>>>>>> > > >>>>>>>> David > > >>>>>>>> > > >>>>>>>> On Wed, May 27, 2026 at 12:44 PM Fabian Hueske < > > [email protected]> > > >>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Thanks Gustavo and Timo for the positive feedback! > > >>>>>>>>> > > >>>>>>>>> I'd like to bump this thread up to collect more feedback. > > >>>>>>>>> If there are no more responses, I will start a vote on this > FLIP > > >>>>> next > > >>>>>>>>> Monday, June 1st. > > >>>>>>>>> > > >>>>>>>>> Best, Fabian > > >>>>>>>>> > > >>>>>>>>> Am Do., 21. Mai 2026 um 12:15 Uhr schrieb Timo Walther < > > >>>>>>>> [email protected] > > >>>>>>>>>> : > > >>>>>>>>> > > >>>>>>>>>> Hi Fabian, > > >>>>>>>>>> > > >>>>>>>>>> thanks for proposing this FLIP. I agree that this join is > super > > >>>>>>> common, > > >>>>>>>>>> after talking to many people at conferences, I could imagine > it > > >>>>>>> will be > > >>>>>>>>>> one of the most used kinds of joins going forward. > > >>>>>>>>>> > > >>>>>>>>>> Tightly coupling it with watermarks fits both from a > semantical > > >>>>>>> point > > >>>>>>>> of > > >>>>>>>>>> view but also with other efforts such as FLIP-558 > (Improvements > > >>>> to > > >>>>>>>>>> SinkUpsertMaterializer and changelog disorder) [1]. In the > near > > >>>>>>> future, > > >>>>>>>>>> we should work on more automated watermarking to power these > > >>>>>>>>>> watermark-based operators, but this is an orthogonal effort. > > >>>>>>>>>> > > >>>>>>>>>> Overall I'm strongly +1 on this. Also +1 on the syntax > > >>>>> improvements > > >>>>>>> for > > >>>>>>>>>> lateral table functions by dropping the TABLE() wrapper. > > >>>>>>>>>> > > >>>>>>>>>> Cheers, > > >>>>>>>>>> Timo > > >>>>>>>>>> > > >>>>>>>>>> [1] > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-558%3A+Improvements+to+SinkUpsertMaterializer+and+changelog+disorder > > >>>>>>>>>> > > >>>>>>>>>> On 18.05.26 11:47, Gustavo de Morais wrote: > > >>>>>>>>>>> Hi Fabian, > > >>>>>>>>>>> > > >>>>>>>>>>> In general a strong +1 for the feature, without getting into > > >>>> the > > >>>>>>>>> details > > >>>>>>>>>> of > > >>>>>>>>>>> the FLIP yet. This is a missing feature for years and I'm > > >>>> happy > > >>>>>>> that > > >>>>>>>>>> we're > > >>>>>>>>>>> putting the time to address this - while also getting rid of > > >>>>> some > > >>>>>>> of > > >>>>>>>>> the > > >>>>>>>>>>> hard restrictions we had. Thanks! > > >>>>>>>>>>> > > >>>>>>>>>>> Kind regards, > > >>>>>>>>>>> Gustavo > > >>>>>>>>>>> > > >>>>>>>>>>> On Fri, 15 May 2026 at 16:39, Fabian Hueske < > > >>>> [email protected] > > >>>>>> > > >>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> Hi everyone, > > >>>>>>>>>>>> > > >>>>>>>>>>>> I'd like to start a discussion on FLIP-579: LATERAL SNAPSHOT > > >>>>> Join > > >>>>>>>> [1]. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Enriching a stream with data from a (slowly changing) > dynamic > > >>>>>>> table > > >>>>>>>>> is a > > >>>>>>>>>>>> super common use case. > > >>>>>>>>>>>> Flink SQL features Temporal Joins [2] to address these use > > >>>>> cases. > > >>>>>>>>>>>> However, SQL users can only use the event-time variant which > > >>>>> has > > >>>>>>>> many > > >>>>>>>>>>>> limitations (heavy dependency on frequent WM updates on both > > >>>>>>> inputs, > > >>>>>>>>>>>> build-side table requires a PK, the join predicate must > > >>>> include > > >>>>>>> the > > >>>>>>>>>>>> build-side PK, etc). > > >>>>>>>>>>>> The processing-time temporal join is disabled (due to > > >>>>> build-side > > >>>>>>>>>>>> initialization issues [3]) and temporal table function joins > > >>>>> are > > >>>>>>>>>>>> only available in Table API. > > >>>>>>>>>>>> > > >>>>>>>>>>>> FLIP-579 proposes a new temporal join operator that operates > > >>>> in > > >>>>>>>>>>>> processing-time and addresses the limitations of the > existing > > >>>>>>>>>>>> implementations: > > >>>>>>>>>>>> * initialization of the build-side before joining > > >>>>>>>>>>>> * no requirement of continuous, frequent build-side WMs > > >>>> (after > > >>>>>>> the > > >>>>>>>>>>>> initialization completed) > > >>>>>>>>>>>> * no requirement for a PK on the build-side > > >>>>>>>>>>>> * table function-based syntax [4] via a built-in SNAPSHOT > > >>>>>>> function > > >>>>>>>>>>>> (proposed in FLIP-517 [4]) > > >>>>>>>>>>>> > > >>>>>>>>>>>> Looking forward to your feedback. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Best, > > >>>>>>>>>>>> Fabian > > >>>>>>>>>>>> > > >>>>>>>>>>>> [1] > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-579%3A+LATERAL+SNAPSHOT+Join > > >>>>>>>>>>>> [2] > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>> > > >> > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sql/queries/joins/#temporal-joins > > >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-19830 > > >>>>>>>>>>>> [4] > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>> > > >> > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sql/queries/joins/#temporal-table-function-join > > >>>>>>>>>>>> [5] > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-517%3A+Better+Handling+of+Dynamic+Table+Primitives+with+PTFs#FLIP517:BetterHandlingofDynamicTablePrimitiveswithPTFs-SNAPSHOTfortemporaljoins > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >> > > >> > > > > >
