Hi, When inserting into PubSub can you set message metadata with the timestamp from the event? If yes then you can make use of:
https://beam.apache.org/releases/javadoc/2.16.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html#withTimestampAttribute-java.lang.String- Cheers Reza On Wed, 9 Oct 2019 at 16:31, Eddy G <[email protected]> wrote: > Thanks a lot for the quick response! > > I can recall having already played with this when I first deployed this > consumer and couldn't get around the following issue that I'm getting now > again... > > java.lang.IllegalArgumentException: Cannot output with timestamp > 2019-10-09T03:12:04.250Z. Output timestamps must be no earlier than the > timestamp of the current input (2019-10-09T03:12:04.292Z) minus the allowed > skew (0 milliseconds). See the DoFn#getAllowedTimestampSkew() Javadoc for > details on changing the allowed skew. > > How can I manage skew? Wouldn't it increase as it's happening with the > current version which uses processing time? > > The timestamp that I'm inferring comes straight from the JSON object > (which is the one looking forward to use) and not from PubSub itself. > -- This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.
