I wanted to mention that elements are not discarded by comparison of their
timestamp with the watermark. They are discarded when their window has
expired - when the watermark is passed the end of the window plus allowed
lateness. An element just has to that arrive for aggregation - such as GBK
or Combine or stateful ParDo - before the window is expired. This drops a
lot less data and is a bit less arbitrary than comparing element timestamp
I wish I had a better answer for how to gain insights into the data being
dropped. That's something we should think about.
On Wed, Feb 7, 2018 at 10:30 PM, Carlos Alonso <car...@mrcalonso.com> wrote:
> Cool, I'll try some of these. Thanks Raghu!
> On Thu, Feb 8, 2018 at 6:40 AM Raghu Angadi <rang...@google.com> wrote:
>> The watermark is not directly available, you essentially infer from fired
>> triggers (e.g. fixed windows). I would consider some of these options :
>> - Adhoc debugging : if the pipeline is close to realtime, you can just
>> guess if a element will be dropped based on its timestamp and current time
>> in the first stage (before first aggregation)
>> - Increase allowed lateness (say to 3 days) and drop the elements
>> yourself you notice are later than 1 day.
>> - Place the elements into another window with larger allowed lateness
>> and log very late elements in another parallel aggregation (see
>> TriggerExample.java in Beam repo).
>> On Wed, Feb 7, 2018 at 2:55 PM, Carlos Alonso <car...@mrcalonso.com>
>>> Hi everyone!!
>>> I have a streaming job running with fixed windows of one hour and
>>> allowed lateness of two days and the number of dropped due to lateness
>>> elements is slowly, but continuously growing and I'd like to understand
>>> which elements are those.
>>> I'd like to get the watermark from inside the job to compare it against
>>> each element and write log messages with the ones that will be potentially
>>> discarded.... Does that approach make any sense? I which case... How can I
>>> get the watermark from inside the job? Any other ideas?
>>> Thanks in advance!!