Re: [PROPOSAL] Prepare Beam 2.8.0 release

+1 On Wed, Oct 3, 2018 at 9:52 AM Jean-Baptiste Onofré wrote: > +1 > > but we have to be fast in release process. 2.7.0 took more than 1 month > to be cut ! > > If no blocker, we have to just move forward. > > Regards > JB > > On 03/10/2018 18:25, Ahmet Altay wrote: > > Hi all, > > > > Release

Re: [apachecon 2018] Universal metrics with apache beam

Very interesting talk, Etienne. Looking forward to the audio recording. Cheers

Re: [DISCUSS] [BEAM-4126] Deleting Maven build files (pom.xml) grace period?

+1 on this effort Original message From: Chamikara Jayalath Date: 6/6/18 2:09 PM (GMT-08:00) To: dev@beam.apache.org, u...@beam.apache.org Subject: Re: [DISCUSS] [BEAM-4126] Deleting Maven build files (pom.xml) grace period? +1 for the overall effort. As Pablo mentioned, we

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

ewritten yet), but it will > be, once more runners catch up with the support: currently we have Dataflow > and Flink. +Chamikara Jayalath <chamik...@google.com> is currently > working on implementing it using SDFs in the Python SDK. > > On Thu, Mar 8, 2018 at 4:34 PM Ted Yu <yuzhih...@gm

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Eugene: Very informative talk. I looked at: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTrackerTest.java Is there some example showing how OffsetRangeTracker works with Kafka partition(s) ? Thanks On Thu, Mar 8, 2018 at 3:58 PM, Eugene Kirpichov

Re: [VOTE] Release 2.3.0, release candidate #3

bq. is OK with staging repo extension ? +1 on using the above approach. Cheers

Re: [DISCUSS] State of the project: Culture and governance

bq. 1. Propose a pull request to their fix branch. This is my favorite and I've mentioned it. Everything is straightforward and explicit. The above makes sense. When the contributor merges the reviewer's pull request, it signifies their willingness to adopt the suggestion, making the combined

Re: Eclipse support

Have you tried running 'mvn eclipse:eclipse' and importing from the root of workspace ? On Wed, Jan 17, 2018 at 10:32 AM, Ron Gonzalez wrote: > Hi, > I've been trying this for a couple of days now, but I can't seem to get > a clean Eclipse import. > I refreshed to

Re: Strata Conference this March 6-8

+1 to BoF On Tue, Jan 16, 2018 at 5:00 PM, Dmitry Demeshchuk wrote: > Probably won't be attending the conference, but totally down for a BoF. > > On Tue, Jan 16, 2018 at 4:58 PM, Holden Karau > wrote: > >> Do interested folks have any timing

Re: [DISCUSS] State of the project

f an issue for many other open-source > projects that do not make such a compatibility guarantee, as they are able > to simply remove or fix the API in the next version. > > I think we still need honest review of PRs, with the criteria being > stronger than "the build doesn

Re: Dataflow runner examples build fail

+1 Original message From: Jean-Baptiste Onofré Date: 1/8/18 1:26 AM (GMT-08:00) To: dev@beam.apache.org Subject: Dataflow runner examples build fail Hi guys, The PRs and nightly builds are failing due to an issue with the dataflow platform: it seems we

Re: [VOTE] Choose the "new" Spark runner

[ ] Use Spark 1 & Spark 2 Support Branch [X] Use Spark 2 Only Branch On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré wrote: > Hi guys, > > To illustrate the current discussion about Spark versions support, you can > take a look on: > > -- > Spark 1 & Spark 2

Re: [VOTE] Release 2.2.0, release candidate #3

Considering that the holiday is around the corner, it would be nice to release 2.2.0 sooner.  Cheers  Original message From: Chamikara Jayalath Date: 11/10/17 12:22 PM (GMT-08:00) To: dev@beam.apache.org Subject: Re: [VOTE] Release 2.2.0,

Re: [VOTE] Drop Spark 1.x support to focus on Spark 2.x

Having both Spark1 and Spark2 modules would benefit wider user base. I would vote for that. Cheers On Wed, Nov 8, 2017 at 12:51 AM, Jean-Baptiste Onofré wrote: > Hi Robert, > > Thanks for your feedback ! > > From an user perspective, with the current state of the PR, the

Re: [DISCUSS] Move away from Apache Maven as build tool

<kirpic...@google.com.INVALID> wrote: > >> >>> > > >> > >> > >Hi! > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >Many of these points sound valid, but AFAICT Maven > >> &g

Re: [Proposal] Apache Beam Swag Store

+1 Original message From: Jacob Marble Date: 11/5/17 1:50 PM (GMT-08:00) To: dev@beam.apache.org Subject: Re: [Proposal] Apache Beam Swag Store I think this is a great idea, ready to order mine. :) Jacob On Sat, Oct 28, 2017 at 11:19 AM, Jean-Baptiste

Re: Policy for stale PRs

ts author fails to respond to actionable comments > for 60 days. The community will close stale PRs. Author is welcome to > reopen the same PR again in the future. The associated JIRAs will be > unassigned from the author but will stay open. > > On Wed, Aug 16, 2017 at 3:25 PM, Ted Yu <yuz

Re: Policy for stale PRs

d, Aug 16, 2017 at 4:32 AM, Aviem Zur <aviem...@gmail.com> > wrote: > >> > > >> > > Makes sense to close after a long time of inactivity and no > response, > >> and > >> > > as Kenn mentioned they can always re

Re: Policy for stale PRs

; as Kenn mentioned they can always re-open. > > On Wed, Aug 16, 2017 at 12:20 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > If we consider the author, it makes sense. > > > > Regards > > JB > > > > On Aug 15, 2017,

Re: [ANNOUNCEMENT] New committers, August 2017 edition!

Congratulations to all. On Fri, Aug 11, 2017 at 10:40 AM, Davor Bonaci wrote: > Please join me and the rest of Beam PMC in welcoming the following > contributors as our newest committers. They have significantly contributed > to the project in different ways, and we look

Re: beam-site issues with Jenkins and MergeBot

However, the following is accessible: https://github.com/apache/beam-site.git Last commit was 13 days ago. On Wed, Aug 9, 2017 at 1:12 PM, Ted Yu <yuzhih...@gmail.com> wrote: > For #1, under https://git-wip-us.apache.org/repos/asf , I don't see > beam-site > > FYI > >

Re: beam-site issues with Jenkins and MergeBot

For #1, under https://git-wip-us.apache.org/repos/asf , I don't see beam-site FYI On Wed, Aug 9, 2017 at 1:08 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Hello, > > I've been trying to merge a PR https://github.com/apache/ > beam-site/pull/278 > and ran into the following

Re: is it ok to have a dicussion without subscribe the list

Derek: You can periodically visit: http://search-hadoop.com/Beam where it is easy to find the thread(s) you're interested in. The latency of indexing is very low. FYI On Wed, Aug 9, 2017 at 1:28 AM, derek wrote: > On Tue, Aug 8, 2017 at 1:23 PM, Jason Kuster

Re: Java 8?

Recently on Spark and Flink mailing lists there have been discussion on dropping support for Java 7. For master branch, we should consider moving to Java 8 for compilation. On Fri, Jul 14, 2017 at 2:41 PM, Swapnil Bawaskar wrote: > Hi, > > I am trying to write a write an

Re: MergeBot is here!

For https://gitbox.apache.org/setup/ , after completing the first two steps, is there any action needed for "MFA Status" box ? Cheers On Fri, Jul 7, 2017 at 1:37 PM, Lukasz Cwik wrote: > for i in range(0, inf): +1 > > Note that the URL for gitbox linking is: >

Re: writing to s3 in beam

Please take a look at BEAM-2500 (and related JIRAs). Cheers On Wed, Jul 5, 2017 at 8:00 PM, Jyotirmoy Sundi wrote: > Hi Folks, > > I am trying to write to s3 from beam. > > These are configs I am passing > > --hdfsConfiguration='[{"fs.default.name": "s3://xxx-output",

Re: Build Failure in * release-2.0.0

bq. Caused by: java.net.SocketException: Too many open files Please adjust ulimit. FYI On Wed, Jul 5, 2017 at 1:33 PM, Jyotirmoy Sundi wrote: > Hi Folks , > > Any idea why the build is failing in release-2.0.0 , i did "mvn clean > package" > > > *Trace* > > [INFO] Running

Re: Build failed in Jenkins: beam_PerformanceTests_Python #47

AddRunnerProfileMvnArgument is called. On Sat, Jul 1, 2017 at 9:19 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > File "<https://builds.apache.org/job/beam_PerformanceTests_ > > Python/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_benchma > rk_helper.py",>

Re: Build failed in Jenkins: beam_PerformanceTests_Python #47

> File " Python/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_ benchmark_helper.py",> > line 136, in InitializeBeamRepo > mvn_command) > TypeError: AddRunnerProfileMvnArgument() takes exactly 3 arguments (2 > given) Looks like 3rd

Re: Java Cross-JDK Test Available on Jenkins Postcommit!

This is nice. Nit: I don't see JDK 1.8 (Oracle) variant. Is this intentional ? Thanks On Fri, Jun 9, 2017 at 3:49 PM, Mark Liu wrote: > Hi all, > > I worked on BEAM-1544 > for > a while, which is to bring Java

Re: Precommit Jenkins Linkage Broken

INFRA-14247 is currently marked Major. Suggest raising the priority so that it gets more attention. Cheers On Tue, May 30, 2017 at 2:59 PM, Jason Kuster < jasonkus...@google.com.invalid> wrote: > Hey folks, > > Just wanted to mention on the dev list that Jenkins precommit breakage is a > known

Re: How can I disable running Python SDK tests when testing my Java change?

think this kind of information would be very useful to speed up new > developers. > To figure this out one has to go through pom files. > > Cheers > > On Fri 19 May 2017 at 18:41, Ted Yu <yuzhih...@gmail.com> wrote: > > > Should this tip be added to https://beam.apa

Re: How can I disable running Python SDK tests when testing my Java change?

Should this tip be added to https://beam.apache.org/contribute/testing/ ? Cheers On Fri, May 19, 2017 at 10:31 AM, Lukasz Cwik wrote: > Developers can use *-pl \!sdks/python* to skip the python module. > > Breaking it up would help developers working outside of Python

Re: [PROPOSAL] design of DSL SQL interface

Can you fill out the Transition Plan ? Thanks On Fri, May 12, 2017 at 10:49 PM, Mingmin Xu wrote: > Hi all, > > As you may know, we're working on BeamSQL to execute SQL queries as a Beam > pipeline. This is a valuable feature, not only shipped as a packaged CLI, > but also

Re: First stable release: version designation?

g3hG53lOYl73tdk > Wqz5B6eQ40TEgk/edit?usp=sharing> > . > I have already added all the votes that Davor compiled 3 hours ago and the > responses afterwards. > > On Thu, May 4, 2017 at 12:49 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > Maybe create a google doc with co

Re: First stable release: version designation?

Mar 8, 2017 at 12:58 PM, Amit Sela <amitsel...@gmail.com> > > > wrote: > > > > > > > > > > > If we were to go with a 2.0 release, we would have to be very > clear > > > on > > > > > > maturity of different modules; for ex

Re: Congratulations Davor!

Congratulations, Davor! On Thu, May 4, 2017 at 12:45 AM, Aviem Zur wrote: > Congrats Davor! :) > > On Thu, May 4, 2017 at 10:42 AM Jean-Baptiste Onofré > wrote: > > > Congrats ! Well deserved ;) > > > > Regards > > JB > > > > On 05/04/2017 09:30 AM, Jason

Re: git commit message

). > > > > However, almost all changes go via PR and most of with a Jira. > > > > So, if maybe you found couple of, again, it's rare and we follow the > > Jira/PR/review/commit path. > > > > Regards > > JB > > > > On 04/30/2017 04:59 PM,

Re: Status of our CI tools

+1 On Fri, Apr 28, 2017 at 6:24 PM, Thomas Groh wrote: > +1! This will be really helpful when looking at my PRs; I basically get no > signal from the current state of the github UI, and this will restore that > to giving me very strong positive signal. > > On Fri, Apr

Re: Community hackathon

+1 > On Apr 24, 2017, at 12:51 AM, Jean-Baptiste Onofré wrote: > > That's a wonderful idea ! > > I think the easiest way to organize this event is using the Slack channels to > discuss, help each other, and sync together. > > Regards > JB > >> On 04/24/2017 09:48 AM,

Re: Hanging Jenkins builds.

Looks like this might be the cause for the failed build ( https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/9927/console ):

Re: [DISCUSSION] Encouraging more contributions

+1 On Sat, Apr 22, 2017 at 7:31 AM, Aviem Zur wrote: > Hi all, > > I wanted to start a discussion about actions we can take to encourage more > contributions to the project. > > A few points I've been thinking of: > > 1. Have people unassign themselves from issues they're

Re: Build failed in Jenkins: beam_SeedJob #214

t I'm giving the evaluate thing a try[1][2] (once it > starts running -- executors are full currently). > > [1] https://builds.apache.org/view/Beam/job/beam_SeedJob/215/ > [2] https://github.com/apache/beam/pull/2578 > > On Tue, Apr 18, 2017 at 10:12 AM, Ted Yu <yuzhih...@g

Re: Build failed in Jenkins: beam_SeedJob #214

;da...@apache.org> wrote: > Not so simple, unfortunately [1]. Ideas welcome ;-) > > Davor > > [1] > https://github.com/jenkinsci/job-dsl-plugin/wiki/Migration# > migrating-to-160 > > On Tue, Apr 18, 2017 at 9:57 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > >

Re: Pipeline termination in the unified Beam model

Why is the timeout needed for Spark ? Thanks > On Apr 18, 2017, at 3:05 AM, Etienne Chauchot wrote: > > +1 on "runners really terminate in a timely manner to easily programmatically > orchestrate Beam pipelines in a portable way, you do need to know whether > the pipeline

Re: Renaming SideOutput

+1 > On Apr 11, 2017, at 5:34 PM, Thomas Groh wrote: > > I think that's a good idea. I would call the outputs of a ParDo the "Main > Output" and "Additional Outputs" - it seems like an easy way to make it > clear that there's one output that is always expected, and

Re: IO ITs: Hosting Docker images

+1 > On Apr 7, 2017, at 10:46 PM, Jean-Baptiste Onofré wrote: > > Hi Stephen, > > I think we should go to 1 and 4: > > 1. Try to use existing images providing what we need. If we don't find > existing image, we can always ask and help other community to provide so. > 4. If

Re: [PROPOSAL]: a new feature branch for SQL DSL

Working in feature branch is good - you may want to periodically sync up with master. I noticed that you are using 1.11.0 of calcite. 1.12 is out, FYI On Wed, Apr 5, 2017 at 2:05 PM, Mingmin Xu wrote: > Hi all, > > I'm working on

Re: [DISCUSSION] Consistent use of loggers

+1 > On Apr 3, 2017, at 11:48 AM, Aviem Zur wrote: > > Upon further inspection there seems to be an issue we may have overlooked: > In cluster mode, some of the runners will have dependencies added directly > to the classpath by the cluster, and since SLF4J can only work

Re: [PROPOSAL] ORC support

+1 > On Apr 1, 2017, at 8:31 AM, Tibor Kiss wrote: > > Hello, > > Recently the Optimized Row Columnar (ORC) file format was spin off from Hive > and became a top level Apache Project: https://orc.apache.org/ > > It is similar to Parquet in a sense that it uses column

Re: Beam spark 2.x runner status

gt; > > I think what we are trying to avoid is to have a long living branch > > > > > for a spark 2 runner based on RDD because the maintenance burden > > > > > would be even worse. We would have to fight not only with the > double > > > > > merge of fixes (in ca

Re: Beam spark 2.x runner status

ent concepts such > as using SparkSession in Spark 2 instead of SparkContext/StreamingContext > in Spark 1. > > On Tue, Mar 21, 2017 at 7:25 PM Ted Yu <yuzhih...@gmail.com> wrote: > > > I have done some work over in HBASE-16179 where compatibility modules are > > c

Re: Beam spark 2.x runner status

I have done some work over in HBASE-16179 where compatibility modules are created to isolate changes in Spark 2.x API so that code in hbase-spark module can be reused. FYI

Re: why Source#validate() is not declared to throw any exception

20, 2017, 11:21 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Hi Ted, > > > > validate() is supposed to throw runtime exception (IllegalStateException, > > RuntimeException, ...) to "traverse" the executor. > > > > Regards > &

why Source#validate() is not declared to throw any exception

Hi, I was reading HDFSFileSource.java where: @Override public void validate() { ... } catch (IOException | InterruptedException e) { throw new RuntimeException(e); } Why is validate() not declared to throw any exception ? If validation doesn't pass, there is nothing to

Re: Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Spark #1282

Looks like Jenkins just came back (to life). Hopefully the next precommit passes. On Sun, Mar 19, 2017 at 11:39 AM, Amit Sela wrote: > I think its a Jenkins issue. Jenkins is shutting down. I'll follow and > relate if this keeps happening. > > -- Forwarded message

Re: [ANNOUNCEMENT] New committers, March 2017 edition!

Congratulations! On Fri, Mar 17, 2017 at 2:13 PM, Davor Bonaci wrote: > Please join me and the rest of Beam PMC in welcoming the following > contributors as our newest committers. They have significantly contributed > to the project in different ways, and we look forward to

Re: [VOTE] Release 0.6.0, release candidate #2

bq. I would prefer that we have a .tar.gz release +1 On Mon, Mar 13, 2017 at 4:21 PM, Ismaël Mejía wrote: > ​+1 (non-binding) > > - verified signatures + checksums > - run mvn clean install -Prelease, all artifacts build and the tests run > smoothly (modulo some local

Re: [VOTE] Release 0.6.0, release candidate #2

I was able to run "mvn clean install -Prelease" command successfully, too. On Sun, Mar 12, 2017 at 12:02 AM, Ahmet Altay wrote: > Amit, > > I was able to successfully build in a clean environment with the following > commands: > > git checkout tags/v0.6.0-RC2 -b RC2 >

Re: [VOTE] Release 0.6.0, release candidate #2

+1 Checked signature. Ran test suite which passed. On Fri, Mar 10, 2017 at 9:05 PM, Ahmet Altay wrote: > Hi everyone, > > Please review and vote on the release candidate #2 for the version 0.6.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve

Re: [VOTE] Release 0.6.0, release candidate #1

bq. ran into a known issue [14] Currently BEAM-1674 is marked blocker. Would it be pushed to next release ? Cheers On Thu, Mar 9, 2017 at 4:07 PM, Ahmet Altay wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version 0.6.0, > as

Re: [VOTE] Release 0.6.0, release candidate #1

+1 Checked signature Ran test suite - all passed. On Thu, Mar 9, 2017 at 4:07 PM, Ahmet Altay wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version 0.6.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the

Re: First stable release: version designation?

If we end up with version 2.0, more effort (trying out more use scenarios e.g.) should go into release process to make sure what is released is indeed stable. Normally people would have higher expectation on 2.0 release compared to 1.0 release. On Mon, Mar 6, 2017 at 6:34 PM, Davor Bonaci

Re: First stable release: version designation?

t for that set of users and > frankly also new ones -- this will make it unambiguous whether referring to > pre-Beam or Beam releases. > > I understand the 1.0 motivation -- it's cleaner in isolation -- but I think > it would lead to long-term confusion in the user community. > > O