[
https://issues.apache.org/jira/browse/BEAM-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417804#comment-17417804
]
Jonathan Hourany edited comment on BEAM-12803 at 9/20/21, 8:28 PM:
-------------------------------------------------------------------
{quote}Is it possible there's a typo somewhere? It looks like your code
correctly references test_tbl everywhere, but the Calcite error says
'test_tabl'.
{quote}
My bad: the typo is in my comment. In code, the exception correctly states
{{test_tbl}}. The test case you mentioned passes so I'm still not sure why my
example isn't. I'm running the example in a notebook, maybe that has something
to do with it?
PR is up! [https://github.com/apache/beam/pull/15539]
For the test cases that were failing: they all seem to be separate issues with
Python 3.9 and aren't directly tied to this fix so I've started creating new
tickets for them specifically (e.g.
https://issues.apache.org/jira/browse/BEAM-12914)
was (Author: jonathan hourany):
{quote}Is it possible there's a typo somewhere? It looks like your code
correctly references test_tbl everywhere, but the Calcite error says
'test_tabl'.
{quote}
My bad: the typo is in my comment. In code, the exception correctly states
{{test_tbl}}. The test case you mentioned passes so I'm still not sure why my
example isn't. I'm running the example in a notebook, maybe that has something
to do with it?
PR is up! [https://github.com/apache/beam/pull/15539]
For the test cases that were failing: they all seem to be separate issues with
Python 3.9 and aren't directly tied to this fix so I've started creating new
tickets for them specifically (e.g.
https://issues.apache.org/jira/browse/BEAM-12914)
> SqlTransform doesn't work on python 3.9
> ---------------------------------------
>
> Key: BEAM-12803
> URL: https://issues.apache.org/jira/browse/BEAM-12803
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: sean teeling
> Assignee: Brian Hulette
> Priority: P2
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Working example below -(Is there no way to paste pre-formatted code into
> jira?!)- (EDIT: I added the appropriate "code" block)
> {code:python}
> import itertools
> import csv
> import io
> import apache_beam as beam
> from apache_beam.dataframe.io import read_csv
> from apache_beam.transforms.sql import SqlTransform
> def parse_csv(val):
> deflower_headers(iterator):
> return itertools.chain([next(iterator).lower()], iterator)
> return csv.DictReader(lower_headers(io.TextIOWrapper(val.open())))
> class BeamTransformBuilder():
> def build(self, pipeline):
> practices = (
> pipeline
> | beam.io.fileio.MatchFiles("data.csv")
> | beam.io.fileio.ReadMatches()
> | beam.Reshuffle()
> | beam.FlatMap(parse_csv)
> | beam.Map(lambda x: beam.Row(id="test-id"))
> | SqlTransform("""
> SELECT
> id
> FROM PCOLLECTION""")
> )
> practices | beam.Map(print)
> def main():
> builder = BeamTransformBuilder()
> with beam.Pipeline('DirectRunner') as p:
> builder.build(p)
> if __name__ == '__main__':
> main()
> {code}
>
> Results in the error:
>
> {code:java}
> File
> "/usr/local/lib/python3.9/site-packages/apache_beam/typehints/schemas.py",
> line 185, in typing_to_runner_api
> element_type = typing_to_runner_api(_get_args(type_)[0])
> IndexError: tuple index out of range
> {code}
>
>
> Tested on Python 3.9.6.
>
> Annoyingly, it is difficult to test this out on other python versions.
> There's no documentation for how to setup a docker container using
> DirectRunner and running it locally. There's barely any documentation on what
> python versions are supported. And using pyenv, and pip install apache-beam
> requires a lot of other downloads that have conflicts when other versions are
> already installed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
