Ok great, thanks Lukasz. I will try turning off the timestamp attribute on
some of these jobs then!
On Thu, Aug 3, 2017 at 10:14 PM, Lukasz Cwik wrote:
> To my knowledge, autoscaling is dependent on how many messages are
> backlogged within Pubsub and independent of the second subscription (the
To my knowledge, autoscaling is dependent on how many messages are
backlogged within Pubsub and independent of the second subscription (the
second subscription is really to compute a better watermark).
On Thu, Aug 3, 2017 at 1:34 PM, wrote:
> Thanks Lukasz that's good to know! It sounds like we
Thanks Lukasz that's good to know! It sounds like we can halve our PubSub costs
then!
Just to clarify, if I were to remove withTimestampAttribute from a job, this
would cause the watermark to always be up to date (processing time) even if the
job starts to lag behind (build up of unacknowledged
You understanding is correct - the data watermark will only matter for
windowing. It will not affect auto-scaling. If the pipeline is not doing
any windowing, triggering, etc then there is no need to pay for the cost of
the second subscription.
On Thu, Aug 3, 2017 at 8:17 AM, Josh wrote:
> Hi al
Hi all,
We've been running a few streaming Beam jobs on Dataflow, where each job is
consuming from PubSub via PubSubIO. Each job does something like this:
PubsubIO.readMessagesWithAttributes()
.withIdAttribute("unique_id")
.withTimestampAttribute("timestamp");
My understa
