This might break the update compatibility for Dataflow streaming pipelines. +Reuven Lax <[email protected]> +Lukasz Cwik <[email protected]>
In other cases, to save update compatibility, we introduced a user option that changes the coder only when the user explicitly asks for an updated feature that requires the new coder. For example, https://github.com/apache/beam/commit/304882caa89afe24150062b959ee915c79e72ab3 Thanks, Cham On Mon, Aug 3, 2020 at 10:00 AM David Janíček <[email protected]> wrote: > Hello everyone, > > I've reported an issue https://issues.apache.org/jira/browse/BEAM-10292 > which is about broken DefaultFilenamePolicy.ParamsCoder behavior. > DefaultFilenamePolicy.ParamsCoder loses information whether > DefaultFilenamePolicy.Params's baseFilename resource is file or directory > on some filesystems, at least on local FS and HDFS. > > After discussion with @dmvk and @lukecwik, we have agreed that the best > solution could be to take the breaking change and use ResourceIdCoder for > encoding/decoding DefaultFilenamePolicy.Params's baseFilename, this way the > file/directory information is preserved. > The solution is implemented in pull request > https://github.com/apache/beam/pull/12050. > > I'd like to ask if there is a consensus on this breaking change. Is > everyone OK with this? > Thanks in advance for answers. > > Best regards, > David >
