Eugene Kirpichov created BEAM-1197: -------------------------------------- Summary: Slowly-changing external data as a side input Key: BEAM-1197 URL: https://issues.apache.org/jira/browse/BEAM-1197 Project: Beam Issue Type: Wish Components: beam-model Reporter: Eugene Kirpichov Assignee: Frances Perry
I've seen repeatedly the following pattern: a user wants to join a PCollection against a slowly-changing external dataset: e.g. a file on GCS, or a Bigtable, etc. Side inputs come to mind, but current side input mechanisms don't allow for something like periodically reloading the side input. The best hacky solution I came up with for one use case is documented here: http://stackoverflow.com/questions/41254028/can-dataflow-sideinput-be-updated-per-window-by-reading-a-gcs-bucket/41271159#41271159 , we need to do better than this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)