Re: Proposal: Add permanent url to community metrics dashboard

2019-07-22 Thread Mikhail Gryzykhin
Thank you for starting this. I'm working on some maintenance on metrics site meanwhile. Looking to disable auth and intializing dashboards from github sources. We might have to look into proper deployment afterwards as well. --Mikhail On Mon, Jul 22, 2019 at 4:11 PM Pablo Estrada wrote: > Hi

Re: Sort Merge Bucket - Action Items

2019-07-22 Thread Robert Bradshaw
On Mon, Jul 22, 2019 at 7:39 PM Eugene Kirpichov wrote: > > On Mon, Jul 22, 2019 at 7:49 AM Robert Bradshaw wrote: >> >> On Mon, Jul 22, 2019 at 4:04 PM Neville Li wrote: >> > >> > Thanks Robert. Agree with the FileIO point. I'll look into it and see what >> > needs to be done. >> > >> >

Re: Jenkins failures / dependency downloads / gradle caching

2019-07-22 Thread Kenneth Knowles
The PR that is causing me the most trouble is https://github.com/apache/beam/pull/9071. But each merge to 2.7.1 took a long time, and many of them I eventually just had to get a gradle build scan not through Jenkins. I am not starting to think it is actually something to do with this PR. It fails

Re: Proposal: Add permanent url to community metrics dashboard

2019-07-22 Thread Pablo Estrada
Hi all, I've filed https://issues.apache.org/jira/browse/INFRA-18786 for this. Thanks! -P. On Thu, Jul 18, 2019 at 1:38 PM Mikhail Gryzykhin wrote: > +1 explicitly > > On Thu, Jul 18, 2019 at 1:46 AM Łukasz Gajowy > wrote: > >> +1 for pushing this forward. The url "metrics.beam.apache.org"

On Auto-creating GCS buckets on behalf of users

2019-07-22 Thread Pablo Estrada
Hello all, I recently worked on a transform to load data into BigQuery by writing files to GCS, and issuing Load File jobs to BQ. I did this for the Python SDK[1]. This option requires the user to provide a GCS bucket to write the files: - If the user provides a bucket to the transform, the

Re: Sort Merge Bucket - Action Items

2019-07-22 Thread Robert Bradshaw
On Fri, Jul 19, 2019 at 5:16 PM Neville Li wrote: > > Forking this thread to discuss action items regarding the change. We can keep > technical discussion in the original thread. > > Background: our SMB POC showed promising performance & cost saving > improvements and we'd like to adopt it for

Re: python precommits failing at head

2019-07-22 Thread Robert Bradshaw
Looks like https://sourceforge.net/p/docutils/bugs/365/ On Sun, Jul 21, 2019 at 11:56 PM Tanay Tummalapalli wrote: > Hi everyone, > > The Python PreCommit from the Jenkins job "beam_PreCommit_Python_Cron" is > failing[1]. The task :sdks:python:docs is failing with this traceback: > > Traceback

Re: python precommits failing at head

2019-07-22 Thread Robert Bradshaw
This was due to a bad release artifact push. This has now been fixed upstream. On Mon, Jul 22, 2019 at 11:00 AM Robert Bradshaw wrote: > > Looks like https://sourceforge.net/p/docutils/bugs/365/ > > On Sun, Jul 21, 2019 at 11:56 PM Tanay Tummalapalli > wrote: >> >> Hi everyone, >> >> The

Re: How to run DynamoDBIOTest?

2019-07-22 Thread Elliotte Rusty Harold
FYI, I killed the job this morning after letting it run over the weeknd. Over 92 hours and no end in sight: <===--> 84% EXECUTING [92h 1m 15s] > IDLE > IDLE > IDLE > IDLE > IDLE > :sdks:java:io:amazon-web-services2:test > 4 tests completed > :sdks:java:io:amazon-web-services:test > 48

Re: Sort Merge Bucket - Action Items

2019-07-22 Thread Neville Li
Thanks Robert. Agree with the FileIO point. I'll look into it and see what needs to be done. Eugene pointed out that we shouldn't build on FileBased{Source,Sink}. So for writes I'll probably build on top of WriteFiles. Read might be a bigger change w.r.t. collocating ordered elements across files

Re: python precommits failing at head

2019-07-22 Thread Tanay Tummalapalli
Thanks @Robert! Python PreCommit is green now! On Mon, Jul 22, 2019 at 5:57 PM Robert Bradshaw wrote: > This was due to a bad release artifact push. This has now been fixed > upstream. > > On Mon, Jul 22, 2019 at 11:00 AM Robert Bradshaw > wrote: > > > > Looks like

Re: Sort Merge Bucket - Action Items

2019-07-22 Thread Robert Bradshaw
On Mon, Jul 22, 2019 at 4:04 PM Neville Li wrote: > > Thanks Robert. Agree with the FileIO point. I'll look into it and see what > needs to be done. > > Eugene pointed out that we shouldn't build on FileBased{Source,Sink}. So for > writes I'll probably build on top of WriteFiles. Meaning it

Re: How to run DynamoDBIOTest?

2019-07-22 Thread Ismaël Mejía
Created https://issues.apache.org/jira/browse/BEAM-7794 to track this issue, any extra info is welcomed in the meantime maybe a good idea to disable the test. On Mon, Jul 22, 2019 at 3:44 PM Elliotte Rusty Harold wrote: > > FYI, I killed the job this morning after letting it run over the >

Re: Sort Merge Bucket - Action Items

2019-07-22 Thread Eugene Kirpichov
On Mon, Jul 22, 2019 at 7:49 AM Robert Bradshaw wrote: > On Mon, Jul 22, 2019 at 4:04 PM Neville Li wrote: > > > > Thanks Robert. Agree with the FileIO point. I'll look into it and see > what needs to be done. > > > > Eugene pointed out that we shouldn't build on FileBased{Source,Sink}. So >

Enhancement for Joining Unbounded PCollections of different WindowFns

2019-07-22 Thread rahul patwari
Hi, Beam currently doesn't support Join of Unbounded PCollections of different WindowFns ( https://beam.apache.org/documentation/programming-guide/#groupbykey-and-unbounded-pcollections ). BeamSql performs [Unbounded PCollection] JOIN [Bounded PCollection], by performing 'SideInputJoin' with