Beam Dependency Check Report (2019-11-11)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue mock 2.0.0 3.0.5 2019-05-20

PR-8814 breaks BigQuery for me

The Pull Request, PR-8814 , breaks my code. BigQuery returns an array whose elements are not represented as lists of maps where each map contains the element value under a key named 'v'. The elements are simple maps, if array of records, or array of base

contributor permission for Beam Jira tickets

Hi, This is Rustam Khalmurzaev (username Rustam_Kh). I'm studying Spark and Beam and trying to use Python SDK. Can someone add me as a contributor for Beam's Jira issue tracker? I would like to create/assign tickets for my work. Thanks, Rustam

Re: [DISCUSS] Avoid redundant encoding and decoding between runner and harness

Thank you all for your feedbacks. @Kenn, @Robert I got what you mean now. I think it's good to make window/timestamp/paneinfo values configurable while also avoid redundant encoding and decoding. Besides, I think the idea from @Luke is good, that we can replace WindowedValue with T in some cases

Re: [Discuss] Beam mascot

9 and 7 for me (in that order) On Mon, Nov 11, 2019 at 7:18 AM Maximilian Michels wrote: > Here are some sketches from the designer. I've put them all in one image > and added labels to make it easier to refer to them. My favorites are > (2) and (9). > > Cheers, > Max > > On 09.11.19 19:43,

Re: Revamping the cross-language validate runner test suite

Thanks for the doc Heejong. I believe this effort will allow us to keep cross-language transforms feature healthy for various SDK combinations by providing clear test signals. Thanks, Cham On Fri, Nov 8, 2019 at 5:27 PM Heejong Lee wrote: > Hi, > > I'm working on revamping the cross-language

Re: Completeness of Beam Java Dependency Check Report

On Mon, Nov 11, 2019 at 10:14 AM Tomo Suzuki wrote: > Hi Beam developers, > (I'm thinking to contribute to upgrades of Java dependencies of Beam; I > just read https://beam.apache.org/contribute/dependencies/) > Thanks that will be great. > > As per the weekly report, Apache Beam Java SDK

Re: Any chance I could help do code reviews in beam

What is your github user id so people could tag you as a reviewer? On Mon, Nov 11, 2019 at 11:02 AM Brandon Pollack wrote: > I might have some off time once and an while and could use a change in > pace sometimes, I wouldn't be able to much but Luke said you guys could use > the help? > > -- >

Re: Questions about the current and future design of the job service message stream

On Sun, Nov 10, 2019 at 5:06 PM Chad Dombrova wrote: > Hi, > >> You can see that each JobMessagesResponse may contain a message *or* a >>> GetJobStateResponse. >>> >>> What’s the intention behind this design? >>> >> I believe this was because a user may want to listen to both job state >> and

Re: Completeness of Beam Java Dependency Check Report

Hi, Thanks for taking care of Beam dependencies. The guava was tracked in BEAM-5559 . It was filtered out by the tool because of the target version is x.y-jre. On the other hand, I checked the logs of dependency job and found that the high

Re: Completeness of Beam Java Dependency Check Report

Chamikara and Yifan, Thank you for the responses! Looking forward to hearing the investigation result. In the meantime, I'll explore .test-infra/jenkins/dependency_check directory.

Re: contributor permission for Beam Jira tickets

Welcome, I have added you as a contributor. On Mon, Nov 11, 2019 at 8:38 AM Рустам Халмурзаев wrote: > Hi, > > This is Rustam Khalmurzaev (username Rustam_Kh). I'm studying Spark and > Beam and trying to use Python SDK. Can someone add me as a contributor for > Beam's Jira issue tracker? I

Completeness of Beam Java Dependency Check Report

Hi Beam developers, (I'm thinking to contribute to upgrades of Java dependencies of Beam; I just read https://beam.apache.org/contribute/dependencies/) As per the weekly report, Apache Beam Java SDK only has 8 outdated dependencies based on the criteria. However, it seems many others are not

Re: PR-8814 breaks BigQuery for me

Could you create a JIRA with steps to reproduce the issue ? Thanks, Cham On Mon, Nov 11, 2019 at 8:29 AM Alan Cabrera wrote: > The Pull Request, PR-8814 , > breaks my code. BigQuery returns an array whose elements are not > represented as lists of maps

Any chance I could help do code reviews in beam

I might have some off time once and an while and could use a change in pace sometimes, I wouldn't be able to much but Luke said you guys could use the help? -- Let me know how I'm doing! go/brpol-feedback ~||~ Flume B B B i a r g l a d n d o n

Re: [Discuss] Beam mascot

I like 9 as well. Not related to anything, but chinchillas are also cute. On Mon, Nov 11, 2019 at 8:25 AM Luke Cwik wrote: > 9 and 7 for me (in that order) > > On Mon, Nov 11, 2019 at 7:18 AM Maximilian Michels wrote: > >> Here are some sketches from the designer. I've put them all in one

Re: Key encodings for state requests

On Fri, Nov 8, 2019 at 10:04 PM jincheng sun wrote: > > > Let us first define what are "standard coders". Usually it should be the > > coders defined in the Proto. However, personally I think the coders defined > > in the Java ModelCoders [1] seems more appropriate. The reason is that for > >

Re: Cython unit test suites running without Cythonized sources

Thank you for spending time on this to clarify it for all of us! Much appreciated. On Sun, Nov 10, 2019 at 3:45 PM Chad Dombrova wrote: > Hi all, > > >> The sdist step creates a package that should be installed into each >> tox environment. If the tox environment has cython when this apache >>

Re: Python Precommit duration pushing 2 hours

On Fri, Nov 8, 2019 at 5:45 PM Ahmet Altay wrote: > > I looked at the log but I could not figure what is causing the timeout > because the gradle scan links are missing. I sampled a few of the successful > jobs, It seems like python 3.7 and python 2 are running 3 tests in serial >

Date/Time Ranges & Protobuf

While crites@ was investigating using protobuf to represent Apache Beam timestamps within the TestStreamEvents, he found out that the well known type google.protobuf.Timestamp doesn't support certain timestamps we were using in our tests (specifically the max timestamp that Apache Beam supports).

Re: Confusing multiple output semantics in Python

I made https://github.com/apache/beam/pull/9954 that explores this. Thanks for the insight, Ning. Internally, we use a different representation. On Thu, Nov 7, 2019 at 2:27 PM Ning Kang wrote: > Hi Sam, > > Thanks for clarifying the accessor to output when building a pipeline. > > Internally,

Re: [Discuss] Beam mascot

The real answer, what cool schwag can we get based upon the mascot. On Mon, Nov 11, 2019 at 2:04 PM Kenneth Knowles wrote: > I'm with Luke on cuttlefish. We can have color changing schwag... > > On Mon, Nov 11, 2019 at 9:57 AM David Cavazos wrote: > >> I like 9 as well. Not related to

Re: Completeness of Beam Java Dependency Check Report

BeamModulePlugin just contains lists of versions to ease coordination across Beam modules, but mostly does not create dependencies. Most of Beam's modules only depend on a few things there. For example Guava is not a core dependency, but here is where it is actually depended upon: $ find . -name

Re: Completeness of Beam Java Dependency Check Report

Kenn, Thank you for the analysis. Although Guava was randomly picked up, it's great learning for me to learn how you analyzed other modules using Guava. On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles wrote: > BeamModulePlugin just contains lists of versions to ease coordination > across Beam

Re: [Discuss] Beam mascot

Dumbo octopus anyone? https://youtu.be/DmqikqvLLLw?t=263 On Mon, Nov 11, 2019 at 2:06 PM Luke Cwik wrote: > The real answer, what cool schwag can we get based upon the mascot. > > On Mon, Nov 11, 2019 at 2:04 PM Kenneth Knowles wrote: > >> I'm with Luke on cuttlefish. We can have color

Re: Python Precommit duration pushing 2 hours

I'm removing the additional interactive test env + suite and add [interactive] dependencies as extra dependencies in tests_require: https://github.com/apache/beam/pull/10068 On Mon, Nov 11, 2019 at 2:15 PM Robert Bradshaw wrote: > On Fri, Nov 8, 2019 at 5:45 PM Ahmet Altay wrote: > > > > I

Re: Any chance I could help do code reviews in beam

That'd be great. go/github so people can find/identify you. On Mon, Nov 11, 2019 at 11:05 AM Luke Cwik wrote: > > What is your github user id so people could tag you as a reviewer? > > On Mon, Nov 11, 2019 at 11:02 AM Brandon Pollack wrote: >> >> I might have some off time once and an while and

Re: [Discuss] Beam mascot

Cuttlefish are cool, but I don't know how recognizable they are, and they don't scream "fast" or "stream-y" or "parallel processing" to me (not that that's a requirement...) I like that firefly, nice working the logo into the trailing beam of light. On Mon, Nov 11, 2019 at 5:03 PM Udi Meiri

Behavior of TimestampCombiner?

Hi, Folks, I am trying to understand the behavior of TimestampCombiner. I have a test like this: 1. class TimestampCombinerTest(unittest.TestCase): 2. 3. def test_combiner_latest(self): 4. """Test TimestampCombiner with LATEST.""" 5. options = PipelineOptions()

Re: Behavior of TimestampCombiner?

The END_OF_WINDOW is indeed 9.99 (or, in Java, 9.999000), but the results for LATEST and EARLIEST should be 9 and 0 respectively. On Mon, Nov 11, 2019 at 5:34 PM Ruoyun Huang wrote: > > Hi, Folks, > > I am trying to understand the behavior of TimestampCombiner. I have a > test like

Re: Date/Time Ranges & Protobuf

The max timestamp, min timestamp, and end of the global window are all performance hacks in my view. Timestamps in beam are really a tagged union: timestamp ::= min | max | end_of_global | actual_time(... some quantitative timestamp ...) with the ordering min < actual_time(...) <