Re: Increase Portable SDK Harness share of memory?

On Thu, Mar 28, 2019 at 9:30 AM Lukasz Cwik wrote: > The intention is that these kinds of hints such as CPU and/or memory > should be embedded in the environment specification that is associated with > the transforms that need resource hints. > > The environment spec is woefully ill prepared as

Re: GSoC Project Proposal

Hi Tanay, thanks for submitting the proposal. I'll try to take a look on Tuesday to give some feedback. I encourage others to take a look as well : ) Best -P. On Thu, Mar 28, 2019 at 11:58 AM Tanay Tummalapalli wrote: > Hi everyone, > > I am applying for Google Summer of Code with Apache Beam

Re: Metrics support on Flink?

On Fri, Mar 29, 2019 at 2:44 PM Ryan Williams wrote: > + Max, Robert, dev@ > > Your assessment is correct Łukasz! We need to get metrics sent over the > job API from the JobServicePipelineResult in the job service to the > "portable harness" that the user is running directly. > One correction:

Re: Metrics support on Flink?

+ Max, Robert, dev@ Your assessment is correct Łukasz! We need to get metrics sent over the job API from the JobServicePipelineResult in the job service to the "portable harness" that the user is running directly. Robert added the RPC for that in #8018 ,

Re: [PROPOSAL] Standardize Gradle structure in Python SDK

I don't use gradle commands for Python development either, because they are slow (no incremental testing). On Fri, Mar 29, 2019 at 9:16 AM Michael Luckey wrote: > > > On Fri, Mar 29, 2019 at 2:31 PM Robert Bradshaw > wrote: > >> On Fri, Mar 29, 2019 at 12:54 PM Michael Luckey >> wrote: >> >

Re: New contributor

Welcome Niklas! On Fri, Mar 29, 2019 at 9:23 AM Ismaël Mejía wrote: > Welcome Niklas! > > On Fri, Mar 29, 2019 at 3:54 PM Guobao Li wrote: > > > > Welcome! > > > > On Wed, Mar 27, 2019 at 11:12 PM Kenneth Knowles wrote: > >> > >> Welcome! > >> > >> On Wed, Mar 27, 2019 at 2:59 PM Mikhail

Re: New contributor

Welcome Niklas! On Fri, Mar 29, 2019 at 3:54 PM Guobao Li wrote: > > Welcome! > > On Wed, Mar 27, 2019 at 11:12 PM Kenneth Knowles wrote: >> >> Welcome! >> >> On Wed, Mar 27, 2019 at 2:59 PM Mikhail Gryzykhin wrote: >>> >>> Welcome Niklas. >>> >>> This is another location with useful resources

Re: [PROPOSAL] Standardize Gradle structure in Python SDK

On Fri, Mar 29, 2019 at 2:31 PM Robert Bradshaw wrote: > On Fri, Mar 29, 2019 at 12:54 PM Michael Luckey > wrote: > > > > Really like the idea of improving here. > > > > Unfortunately, I haven't worked with python on that scale yet, so bear > with my naive understandings in this regard. If I

Re: Flink 1.7.2 + Beam 2.11 error: The transform beam:transform:create_view:v1 is currently not supported.

Great. Here is the JIRA issue including a PR which fixes the problem and adds additional tests to prevent this in the future: https://issues.apache.org/jira/browse/BEAM-6937 Thanks, Max On 29.03.19 15:43, Kaymak, Tobias wrote: Can confirm that this is the issue, starting with streaming=True

Re: New contributor

Welcome! On Wed, Mar 27, 2019 at 11:12 PM Kenneth Knowles wrote: > Welcome! > > On Wed, Mar 27, 2019 at 2:59 PM Mikhail Gryzykhin > wrote: > >> Welcome Niklas. >> >> This is another location with useful resources for contributors: >>

Re: Flink 1.7.2 + Beam 2.11 error: The transform beam:transform:create_view:v1 is currently not supported.

Can confirm that this is the issue, starting with streaming=True fixes it. On Fri, Mar 29, 2019 at 11:53 AM Maximilian Michels wrote: > Hi Tobias, > > Thank for reporting. Can confirm, this is a regression with the > detection of the execution mode. Everything should work fine if you set > the

Re: Unexpected TestStream behavior when testing stateful DoFn

This reminds me of a bug I had filed for the direct runner a few weeks ago, except I was running into it in Java and not python. https://issues.apache.org/jira/browse/BEAM-6813 On Thu, Mar 28, 2019 at 9:04 PM Ahmet Altay wrote: > I agree it looks like a bug. Filed >

Re: [PROPOSAL] Standardize Gradle structure in Python SDK

On Fri, Mar 29, 2019 at 12:54 PM Michael Luckey wrote: > > Really like the idea of improving here. > > Unfortunately, I haven't worked with python on that scale yet, so bear with > my naive understandings in this regard. If I understand correctly, the > suggestion will result in a couple of

Re: [PROPOSAL] Standardize Gradle structure in Python SDK

Really like the idea of improving here. Unfortunately, I haven't worked with python on that scale yet, so bear with my naive understandings in this regard. If I understand correctly, the suggestion will result in a couple of projects consisting only of a build,gradle file to kind of workaround on

Re: Flink 1.7.2 + Beam 2.11 error: The transform beam:transform:create_view:v1 is currently not supported.

Hi Tobias, Thank for reporting. Can confirm, this is a regression with the detection of the execution mode. Everything should work fine if you set the "streaming" flag to true. Will be fixed for the 2.12.0 release. Thanks, Max On 28.03.19 17:28, Lukasz Cwik wrote: +dev

Re: Python SDK Arrow Integrations

First off, huge +1 to a good integration with Arrow and Beam. I think to fully realize the benefits we need to have deeper integration than arrow-frame-batches as elements, i.e. SDKs should be augmented to understand arrow frames as batches of individual elements, each with (possibly) their own