In the streams/tables way of talking, side inputs are tables. External KV stores are basically also [globally windowed] tables. Both are time-varying.
I think it makes perfect sense to access an external KV store in userland directly rather than listen to its changelog and reproduce the same table as a multimap side input. I'm sure many users are already doing this. I'm sure users will always do this. Providing a common interface (simpler than Filesystem) and helpful transform(s) in an extension module seems nice. Does it require any support in the core SDK? If I understand, Luke & Robert, you favor adding metadata to Read/SDF so that a user _does_ write it as a changelog listener that is observed as a multimap side input, and each runner optimizes it if they can to just directly access the KV store? A runner is free to use any kind of storage they like to materialize a side input anyhow, so this is surely possible, but it is a "sufficiently smart compiler" issue. As for semantics, I'm not worried about availability - it is globally windowed and always available. But I think this requires retractions to be correctly equivalent to direct access. I think we can have a userland PTransform in much less time than a model concept, so I favor it. Kenn
