Re: Support ZetaSQL as a new SQL dialect in BeamSQL

Thank you Rui for the heads up. A question to the community, does the size of the change require any process besides the usual PR reviews? On Fri, Aug 2, 2019 at 10:23 AM Rui Wang wrote: > Hi community, > > I have been working on supporting ZetaSQL[1] as a SQL dialect in BeamSQL. > ZetaSQL is

Re: [BEAM-7819] -python - parsing message_id from PubSub message to the PubSubMessage attributes property

On Wed, Jul 31, 2019 at 4:19 AM Matthew Darwin < matthew.dar...@carfinance247.co.uk> wrote: > Hi Ahmet/Udi, > > There are a couple of additional tests that failed following my change; > apache_beam.io.gcp.tests.pubsub_match_test.PubSubMatcherTest.test_message_matcher_strip_success > and >

Re: [PROPOSAL] An initial Schema API in Python

To clarify, I am happy to start with implementation and iterating on it. I do not want to block this late into the discussion. On Fri, Aug 2, 2019 at 6:03 PM Brian Hulette wrote: > I meant "or sub-class it and define fields with type annotations" not > "with attributes". I believe that version

Re: [PROPOSAL] An initial Schema API in Python

I meant "or sub-class it and define fields with type annotations" not "with attributes". I believe that version doesn't work in python 2 since it doesn't support the `name: type` syntax. On Fri, Aug 2, 2019 at 5:55 PM Brian Hulette wrote: > > Do we need to support python 2? If supporting python

Re: [PROPOSAL] An initial Schema API in Python

> Do we need to support python 2? If supporting python 2 will complicate things, we could make this a python3 only feature. I don't think supporting python 2 complicates things. It's just that there are two different ways to use typing.NamedTuple in python 3 - you can either instantiate it and

Re: [DISCUSS] Dependency management for Python SDK Container

On Fri, Aug 2, 2019 at 4:34 PM Brian Hulette wrote: > Thanks for the reply, I added some responses inline. > > On Fri, Aug 2, 2019 at 2:42 PM Ahmet Altay wrote: > > > > There is a value in explicitly pinning the dependencies to be used in > the containers: > > - It reproducibly produces the

Re: [PROPOSAL] An initial Schema API in Python

Thank you Brian. I did not spend enough time yet to review. Some early questions, I apologize if I missed an earlier discussion. - Do we need to support python 2? If supporting python 2 will complicate things, we could make this a python3 only feature. - Why are we mapping to numpy types? Design

[DISCUSS] Dependency management for Python SDK Container

I recently ran into a portable python precommit failure that led me to discover that python dependencies for the container are defined in two different places, in slightly different ways: in setup.py with version ranges [1], and in a base_image_requirements.txt file in the container directory with

Re: Update on Python Gradle test configuration unification

Thank you, Mark. Everyone, if you have recently added, or are planning to add, a Python SDK test suite, please make sure to replicate this suite to run under at least one Python 3 minor version to make sure use cases work on Python 3. If you pick only one Python 3 minor version (to save resources

Update on Python Gradle test configuration unification

Hi all, Some refactors are made recently on Python Gradle configurations to unify tests among Py2 and Py3x. More specifically, Py2 tests defined in sdks/python/build.gradle were moved to sdks/python/test-suites

Re: Docker Run Options in SDK Container

Hi all, I’m a bit confused about the desire to use json for the environment_config. It’s harder to use json on the command line, such that now we’re talking about the value being *either* a docker image name *or* a path to a json file (OR maybe yaml too!), which is not only less convenient than

Re: [ANNOUNCE] Beam 2.14.0 Released!

Thanks for the release work, Anton! On Fri, Aug 2, 2019 at 9:08 AM Hannah Jiang wrote: > Thanks Anton for all the work to release it. > > On Fri, Aug 2, 2019 at 7:12 AM Connell O'Callaghan > wrote: > >> Well done Anton and all involved!!! >> >> On Fri, Aug 2, 2019 at 06:56 Robert Bradshaw

Support ZetaSQL as a new SQL dialect in BeamSQL

Hi community, I have been working on supporting ZetaSQL[1] as a SQL dialect in BeamSQL. ZetaSQL is a SQL analyzer open sourced by Google. Here is ZetaSQL's documentation[2]. Birfely, the design of integrating ZetaSQL with BeamSQL is, I made a plugable query planner interface in BeamSQL, and we

Re: Query about JdbcIO.readRows()

Agree. I will fix that. Regards JB Le 2 août 2019 à 17:15, à 17:15, Vishwas Bm a écrit: >Hi Kishor, > >+ dev (dev@beam.apache.org) > >This looks like a bug. The attribute statementPreparator is nullable >It should have been handled in the same way as in the expand method of >Read class. > >

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

Hi all, Thanks a lot for sharing your thoughts! It seems that we have already reached consensus for the following items. Could you please read through them again and double-check if you all agree with these? If yes, then I would start creating JIRA issues for those that don’t yet have a JIRA

Re: [ANNOUNCE] Beam 2.14.0 Released!

Thanks Anton for all the work to release it. On Fri, Aug 2, 2019 at 7:12 AM Connell O'Callaghan wrote: > Well done Anton and all involved!!! > > On Fri, Aug 2, 2019 at 06:56 Robert Bradshaw wrote: > >> Lots of improvements all around. Thank you for pushing this through, >> Anton! >> >> On Fri,

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

Thanks for share the detail of the current StandardCoders Max! That's true, Flink may should defined some of coders, And I will share the POC in the Flink Python UDFs DISCUSS Thread later :) Best, Jincheng Maximilian Michels 于2019年7月31日周三 下午2:53写道： > Hi Jincheng, > > Thanks for getting back to

Re: [ANNOUNCE] Beam 2.14.0 Released!

Lots of improvements all around. Thank you for pushing this through, Anton! On Fri, Aug 2, 2019 at 1:37 AM Chad Dombrova wrote: > > Nice work all round! I love the release blog format with the highlights and > links to issues. > > -chad > > > On Thu, Aug 1, 2019 at 4:23 PM Anton Kedin wrote: