[jira] [Updated] (BEAM-945) BigQueryIO.parseTableSpec is package-private
[ https://issues.apache.org/jira/browse/BEAM-945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Martin updated BEAM-945: --- Description: It was changed in this commit. https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6. Is there a good reason for it to not be public? If not I will submit a PR to change it back. was: It was changed in this commit. https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6. Is there a good reason for it to not be public? > BigQueryIO.parseTableSpec is package-private > > > Key: BEAM-945 > URL: https://issues.apache.org/jira/browse/BEAM-945 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp >Affects Versions: 0.3.0-incubating >Reporter: Andrew Martin >Assignee: Daniel Halperin >Priority: Blocker > > It was changed in this commit. > https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6. > Is there a good reason for it to not be public? If not I will submit a PR to > change it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-945) BigQueryIO.parseTableSpec is package-private
Andrew Martin created BEAM-945: -- Summary: BigQueryIO.parseTableSpec is package-private Key: BEAM-945 URL: https://issues.apache.org/jira/browse/BEAM-945 Project: Beam Issue Type: Bug Components: sdk-java-gcp Affects Versions: 0.3.0-incubating Reporter: Andrew Martin Assignee: Daniel Halperin Priority: Blocker It was changed in this commit. https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6. Is there a good reason for it to not be public? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable
[ https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Martin closed BEAM-741. -- Resolution: Not A Problem > Values transform does not use the correct output coder when values is an > Iterable > > > Key: BEAM-741 > URL: https://issues.apache.org/jira/browse/BEAM-741 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Andrew Martin >Assignee: Davor Bonaci > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable
[ https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575937#comment-15575937 ] Andrew Martin commented on BEAM-741: [~kenn] I dug into this more today, and found the specific reason for this failure - the inference process in Beam checks the coder registry, and if it doesn't find any will try to use a fallback coder provider. If it fails there, only then will it try to obtain the coder from the producing transform. In Scio we set our own fallback coder provider, so Beam will never end up using the output coder from the producing transform. So, in Scio we probably need to prefer using the default output coder of the producing transform, and fall back as a last resort. I will close this because it is an issue in Scio, not in Beam. Thanks! > Values transform does not use the correct output coder when values is an > Iterable > > > Key: BEAM-741 > URL: https://issues.apache.org/jira/browse/BEAM-741 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Andrew Martin >Assignee: Davor Bonaci > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable
[ https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Martin reopened BEAM-741: > Values transform does not use the correct output coder when values is an > Iterable > > > Key: BEAM-741 > URL: https://issues.apache.org/jira/browse/BEAM-741 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Andrew Martin >Assignee: Davor Bonaci > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable
[ https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572443#comment-15572443 ] Andrew Martin commented on BEAM-741: [~kenn] After investigating further, it seems like the coder for the output of the values transform is not inferred correctly because of what appears to be some loss of type information in the type descriptor - the output of the `Values` transform should be of type Iterable but the raw type is just Object during the inference process, so the default coder provider is used (which we set in our own code). I'm part of a team at Spotify developing Scio (https://github.com/spotify/scio) and we have a work-in-progress branch for beam porting, and it is some tests in there that fail. I'd like to have a failing test written in the pure beam API so you can take a look - that being said, is it possible to invoke the @RunnableOnService tests locally using the direct runner? > Values transform does not use the correct output coder when values is an > Iterable > > > Key: BEAM-741 > URL: https://issues.apache.org/jira/browse/BEAM-741 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Andrew Martin >Assignee: Davor Bonaci > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable
[ https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1554#comment-1554 ] Andrew Martin commented on BEAM-741: I see, so being explicit about setting the coder in Values is probably a duct-tape solution for a more fundamental problem we have, perhaps something to do with the coder registry (we have some custom coders so the problem is likely here). I will close this. > Values transform does not use the correct output coder when values is an > Iterable > > > Key: BEAM-741 > URL: https://issues.apache.org/jira/browse/BEAM-741 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Andrew Martin >Assignee: Davor Bonaci > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable
[ https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Martin closed BEAM-741. -- Resolution: Not A Problem Fix Version/s: Not applicable > Values transform does not use the correct output coder when values is an > Iterable > > > Key: BEAM-741 > URL: https://issues.apache.org/jira/browse/BEAM-741 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Andrew Martin >Assignee: Davor Bonaci > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable
[ https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566259#comment-15566259 ] Andrew Martin commented on BEAM-741: I found this issue when using the Direct Runner for a test, which does dynamic re-sharding as part of the Write transform. It does a GroupByKey -> Values -> Flatten, and Flatten failed because it did not have an 'IterableLikeCoder' as the input coder, which should be the case after calling Values form a GBK. > Values transform does not use the correct output coder when values is an > Iterable > > > Key: BEAM-741 > URL: https://issues.apache.org/jira/browse/BEAM-741 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Andrew Martin >Assignee: Davor Bonaci > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable
Andrew Martin created BEAM-741: -- Summary: Values transform does not use the correct output coder when values is an Iterable Key: BEAM-741 URL: https://issues.apache.org/jira/browse/BEAM-741 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Andrew Martin Assignee: Davor Bonaci -- This message was sent by Atlassian JIRA (v6.3.4#6332)
