[ https://issues.apache.org/jira/browse/BEAM-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629704#comment-15629704 ]
Daniel Halperin commented on BEAM-881: -------------------------------------- If it is a required part of every IO, aka a change to the IO spec, it's an invasive change. If it's a separate module that has a sole author that produces these companion transforms as desired for IOs, fine. But it will be a separate module, not part of the core. In other words, the title "Provide a PTransform in IOs" needs to be changed to "provide a library for standard interoperability that supports every IO". > Provide a PTransform in IOs providing a "standard" Avro IndexedRecord > --------------------------------------------------------------------- > > Key: BEAM-881 > URL: https://issues.apache.org/jira/browse/BEAM-881 > Project: Beam > Issue Type: New Feature > Components: sdk-java-extensions > Reporter: Jean-Baptiste Onofré > Assignee: Jean-Baptiste Onofré > > Now, each IO is using a different data format. For instance, the > {{JmsIO.Read}} provides a {{PCollection}} of {{JmsRecord}} (and > {{JmsIO.Write}} expects also a {{JmsRecord}}), {{KafkaIO.Read}} provides a > {{PCollection}} of {{KafkaRecord}}. > It could appear a bit "complex" for users to manipulate such kind of data > format: some users may expect kind of standard format. > Without modifying the existing IO, we could add a {{PTransform}} (as part of > the IO) that an user can optionally use. This transform will convert the IO > data format (let say {{JmsRecord}} for instance) to a standard Avro > {{IndexedRecord}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)