[
https://issues.apache.org/jira/browse/BEAM-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724274#comment-16724274
]
Scott Wegner commented on BEAM-6261:
------------------------------------
I spoke with [[email protected]] who mentioned this is a bug in the Dataflow
service; the backend is not properly expiring cached side input values.
A fix is in progress and when it rolls out will apply to all SDK versions.
> Dataflow runner does not refresh updated side inputs
> ----------------------------------------------------
>
> Key: BEAM-6261
> URL: https://issues.apache.org/jira/browse/BEAM-6261
> Project: Beam
> Issue Type: New Feature
> Components: runner-dataflow
> Reporter: Scott Wegner
> Assignee: Daniel Mills
> Priority: Major
>
> See [this user@
> thread|https://lists.apache.org/thread.html/5eed0fc3beeb9f1c1fe4a623cbcad41cb15d0d80490cafb1f27e4577@%3Cuser.beam.apache.org%3E].
> The [Slowly-changing lookup
> cache|https://cloud.google.com/blog/products/gcp/guide-to-common-cloud-dataflow-use-case-patterns-part-1]
> pattern described on the GCP blog uses a side input to feed lookup data to
> join with the main data input. However, the Dataflow runner doesn't update
> side inputs.
> Example pipeline here: https://github.com/lbordwell/sideinput
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
