Re: [ACTION REQUESTED] Triage 3-10 issues (get the labels right, basically)

I closed several issues that have already been fixed. It looks we didn’t reference the issue number in commit message which can auto-close the issue. Kenneth Knowles 于2022年10月31日 周一20:01写道： > Hi all > > We have built up to 138 GitHub Issues with the "awaiting triage" label. If > you can, take jus

Re: What to do about issues that track flaky tests?

Agreed. I also mentioned in a previous email that some issues have been open for a long time (before being migrated to GitHub) and it's possible that those tests can pass constantly now. We may double check and close them since reopening is just one click. Manu On Thu, Sep 15, 2022 at 6:58 AM Aus

Re: Beam High Priority Issue Report

I find some issues were very old before the migration from JIRA and have no updates after the migration. Is there a stale bot to downgrade them? Meanwhile, would it be better to have "affected versions" in the issue template? On Fri, Jul 8, 2022 at 6:03 PM wrote: > This is your daily summary of

Re: [DISCUSS] What to do about P0/P1/flake automation Was: P1 issues report (70)

urrent automation in any form >>>>> we should fix the url from >>>>> https://api.github.com/repos/apache/beam/issues/# to >>>>> https://github.com/apache/beam/issues/# - the current links are very >>>>> annoying. >>>>> - Afte

Re: P1 issues report (70)

Hi all, what is this daily summary intended for? Not all issues look like P1. And will a weekly summary be less noise? 于2022年6月22日 周三23:45写道： > This is your daily summary of Beam's current P1 issues, not including > flaky tests. > > See https://beam.apache.org/contribute/issue-priorities/#p1

Re: Beam Metrics Report (2022-04-05)

Is this still working? Apache Jenkins Server 于2022年4月5日 周二19:56写道： > ERROR: File > 'src/.test-infra/jenkins/metrics_report/beam-metrics_report.html' does not > exist

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[] Beaver [] Hedgehog [x] Lemur [x] Owl [ ] Salmon [ ] Trout [ ] Robot dinosaur [ ] Firefly [ ] Cuttlefish [ ] Dumbo Octopus [ ] Angler fish On Wed, Nov 20, 2019 at 8:38 PM Maximilian Michels wrote: > [ ] Beaver > [ ] Hedgehog > [x] Lemur > [ ] Owl > [ ] Salmon > [ ] Trout > [ ] Robot dinosaur >

Re: [DISCUSS] JIRA open/close emails?

+1. Like the subscribe to only "close issue" action instead of each comment on GitHub Manu On Thu, Oct 10, 2019 at 10:51 AM Kenneth Knowles wrote: > Currently, all email from JIRA goes to iss...@beam.apache.org. > > I just learned that HBase has a JIRA configuration so that issue > open/close g

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett wrote: > +1 > > On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev > wrote: > >> Hi everyone, >> >> Please vote whether to sign a pledge on behalf of Apache Beam to sunset >> Beam Python 2 offering (in new releases) in 2020 on >> http://python3s

Re: [VOTE] Support ZetaSQL as another SQL dialect for BeamSQL in Beam repo

+1 On Tue, Aug 13, 2019 at 11:55 AM Mingmin Xu wrote: > +1 > > On Mon, Aug 12, 2019 at 8:53 PM Ryan McDowell > wrote: > >> +1 >> >> On Mon, Aug 12, 2019 at 8:30 PM Reza Rokni wrote: >> >>> +1 >>> >>> On Tue, 13 Aug 2019 at 09:28, Ahmet Altay wrote: >>> +1 On Mon, Aug 12, 2019 a

Re: Support ZetaSQL as a new SQL dialect in BeamSQL

> > A question to the community, does the size of the change require any > process besides the usual PR reviews? > I think so. This is a big change and has come as kind of a surprise (sorry if I've missed previous discussions). Rui, could you explain more on how things will play out between BeamS

Re: Jenkins failures / dependency downloads / gradle caching

ally just had to get a gradle build > scan not through Jenkins. I am not starting to think it is actually > something to do with this PR. It fails in a number of different ways but > always fails. > > On Sun, Jul 21, 2019 at 1:27 AM Manu Zhang > wrote: > >> Hi Kenn, >

Re: [PROPOSAL] Revised streaming extensions for Beam SQL

Thanks Kenn, great paper and left some newbie questions on the proposal. Manu On Fri, Jul 19, 2019 at 1:51 AM Kenneth Knowles wrote: > Hi all, > > I recently had the great privilege to work with others from Beam plus > Calcite and Flink SQL contributors to build a new and minimal proposal for >

Re: Jenkins failures / dependency downloads / gradle caching

Hi Kenn, Could you send the command you've been trying and the error logs ? Manu On Sat, Jul 20, 2019 at 4:33 AM Kenneth Knowles wrote: > Hi all, > > Are we breaking the local maven central cache? I've been unable to get a > green Java PreCommit and it appears to be entirely having trouble > d

Re: Looping timer blog

Indeed interesting pattern. One minor question. It seems the timer is triggered by the first element so what if there is no data in the "first interval" ? Thanks for the write-up. Manu On Wed, Jun 19, 2019 at 12:15 PM Reza Rokni wrote: > Hi folks, > > Just wanted to drop a note here on a new p

Re: [VOTE] Remove deprecated Java Reference Runner code from repository.

+1 On Wed, May 15, 2019 at 7:57 PM Maximilian Michels wrote: > +1 > > On 15.05.19 13:19, Robert Bradshaw wrote: > > +1 for removing the code given the current state of things. > > > > On Wed, May 15, 2019 at 12:32 AM Ruoyun Huang wrote: > >> > >> +1 > >> > >> From: Daniel Oliveira > >> Date: T

Re: [spark runner based on dataset POC] your opinion

; and has many subclasses. Maybe bytes serialization is still the best way to > go, but we don't leverage schema PCollections. > Best > Etienne > > Le jeudi 17 janvier 2019 à 21:52 +0800, Manu Zhang a écrit : > > Nice Try, Etienne ! Is it possible to pass in the schema thro

Re: [spark runner based on dataset POC] your opinion

Nice Try, Etienne ! Is it possible to pass in the schema through pipeline options ? Manu On Thu, Jan 17, 2019 at 5:25 PM Etienne Chauchot wrote: > Hi Kenn, > > Sure, in spark DataSourceV2 providing a schema is mandatory: > - if I set it to null, I obviously get a NPE > - if I set it empty: I ge

Re: gradle clean causes long-running python installs

I have the same question. Sometimes even `./gradlew clean` fails due to failure of `setupVirtualEnv` tasks. Manu Zhang On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles , wrote: > A global `./gradlew clean` runs various `setupVirtualEnv` tasks that invoke > things such as `setup.py bdist

Re: Python 3: final step

b.com/tensorflow/transform/issues/1) which is blocked by BEAM-1251. Thanks, Manu Zhang On Fri, Oct 12, 2018 at 3:17 AM Valentyn Tymofieiev wrote: > I cc'ed a few folks who are familiar with Jenkins setup on > https://issues.apache.org/jira/browse/BEAM-5663, I think we can continue > t

Re: [Fwd: [Apache Beam] Custom DataSourceV2 instanciation: parameters passing and Encoders]

Etienne, Have you looked at   https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Source.scala  ? It seems that’s how Spark integrates Kafka with DataFrame API. Thanks, Manu Zhang On Dec 20, 2018, 5:31 PM +0800, Etienne Chauchot , wrote

Re: Recordings and presentations from Beam Summit London 2018

Thanks for sharing. The YouTube channel link is  https://www.youtube.com/channel/UChNnb_YO_7B0HlW6FhAXZZQ Thanks, Manu Zhang On Nov 2, 2018, 11:06 PM +0800, Matthias Baetens , wrote: > Hi everyone, > > Very happy to be sharing the great materials from the speakers and > contributors

Re: [Fwd: [Apache Beam] Custom DataSourceV2 instanciation: parameters passing and Encoders]

DataSourceV2 ? Thanks, Manu Zhang On Dec 19, 2018, 6:40 PM +0800, Etienne Chauchot , wrote: > Thanks Kenn for taking the time to take a look > > Le mardi 18 décembre 2018 à 11:39 -0500, Kenneth Knowles a écrit : > > I don't know DataSourceV2 well, but I am reading around to try

Re: Doc on running validatesRunner test

Thanks Thomas. We’ve documented commands for `needsRunner` test at  https://beam.apache.org/contribute/testing/  but not for `validatesRunner`.  Do you or anyone else know why they are not included ? Thanks, Manu Zhang On Dec 15, 2018, 12:33 PM +0800, Thomas Weise , wrote: > Thanks for bring

Doc on running validatesRunner test

./gradlew :beam-runners-gearpump:validatesRunnerStreaming —tests org.apache.beam.sdk.PipelineTest Has this been documented anywhere ? Thanks, Manu Zhang

Re: [VOTE] Mark 2.7.0 branch as a long term support (LTS) branch

+1 Thanks, Manu Zhang On Nov 10, 2018, 12:55 PM +0800, Kenneth Knowles , wrote: > +1 > > > On Fri, Nov 9, 2018, 12:04 Scott Wegner > > +1 Thanks for driving this Ahmet! > > > > > > > On Fri, Nov 9, 2018 at 9:33 AM Chamikara Jayalath > > > &

Re: Python profiling

Cool ! Can we document it somewhere such that other Runners could pick it up later ? Thanks, Manu Zhang On Oct 29, 2018, 5:54 PM +0800, Maximilian Michels , wrote: > This looks very helpful for debugging performance of portable pipelines. > Great work! > > Enabling local directories

Re: Python 3: final step

Does anyone know how to set up python version on Jenkins ? It’s Python 3.5.2 now. Thanks, Manu Zhang On Oct 5, 2018, 9:24 AM +0800, Valentyn Tymofieiev , wrote: > I have put together a guide [1] to help get started with investigating Python > 3-related test failures that may be helpful f

Re: Python 3: final step

Thanks Valentyn. Note some test failing issues are covered by “Finish Python 3 porting for *** module”, e.g. https://issues.apache.org/jira/browse/BEAM-5315. Manu 在 2018年10月3日 +0800 PM4:18，Valentyn Tymofieiev ，写道： > Hi Rakesh and Manu, > > Thanks to both of you for offering help (in different thr

Re: Python typing library is not provisional in Python 3.7

-D/view/Beam/job/beam_PostCommit_Python_Verify/6114/consoleFull > > > [2]  > > > https://lists.apache.org/thread.html/5371469de567357b1431606f766217ef73a9098dc45046f51a6ecceb@%3Cdev.beam.apache.org%3E > > > > > > > On Thu, Sep 27, 2018 at 10:09 PM, Manu Zhang &

Re: Python typing library is not provisional in Python 3.7

> >> Hope this helps. >> >> Ahmet >> >> [1] >> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/6114/consoleFull >> [2] >> https://lists.apache.org/thread.html/5371469de567357b1431606f766217ef73a9098dc45046f51a6ecceb@%3Cd

Re: Python typing library is not provisional in Python 3.7

lob/5d298db4c20bbb8876a5b75142341332c1e3fb8d/sdks/python/tox.ini#L56 > [2] > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail > > > On Thu, Sep 27, 2018 at 3:52 PM Manu Zhang > wrote: > >> Hi all, >> >> I failed to run Py

Python typing library is not provisional in Python 3.7

teError: type object 'Callable' has no attribute '_abc_registry' This is because the required typing library is not provisional in Python 3.7 <https://github.com/python/typing#important-dates>. Any thoughts on this? Shall we add Python 3.7 environment to our tests ? Thanks, Manu Zhang

Re: Retiring runners?

I agree we should revisit this question if the runner is too much of a burden. Thanks, Manu Zhang On Fri, Sep 21, 2018 at 6:34 AM Lukasz Cwik wrote: > Datatorrent the largest contributor to the Apex project has closed > shop[1], the project hasn't retired but development has signif

Re: Run Python3 tests in miniconda

the issue with setup? > > I have not tried Miniconda with Beam myself. Perhaps you could describe > your setup in more detail, so that I (or other folks on the list) could try > to reproduce the issue? Also, did you use Miniconda3? > > Thanks, > Valentyn > > > On Wed, Se

Run Python3 tests in miniconda

tible here. Thanks, Manu Zhang

[FYI] Paper of Building Beam Runner for IBM Streams

7;ve) implemented the Gearpump runner. I believe it will be beneficial to have more such papers and discussions as sharing design choices and lessons from various runners. Enjoy it ! Manu Zhang

Re: Is Gearpump post-commit tests jenkins job still valid?

Thanks Mik and Kenn for taking care of this, which should really be my job. On Fri, Jun 22, 2018 at 1:27 AM Mikhail Gryzykhin wrote: > Thank you for update on this. > > PR is up: > https://github.com/apache/beam/pull/5721 > > --Mikhail > > > On Thu, Jun 21, 2018 at 10:12 AM Kenneth Knowles wrot

Re: [VOTE] Use Gradle for Apache Beam developmental processes

+1 (binding) On Wed, Nov 29, 2017 at 5:08 AM Reuven Lax wrote: > +1 (binding) > > One caveat to the second part of this vote. I think we need to elaborate a > clear list of criteria that Gradle must clear before any processes are > migrated off of Maven. > > On Tue, Nov 28, 2017 at 12:51 PM, Ro

Re: [Build] Enforce failed on project beam-sdks-java-io-tika

lternatively, try using the gradle build (from root of project directory): > ./gradlew build > > > > On Fri, Nov 24, 2017 at 9:46 PM, Manu Zhang > wrote: > > > Hi all, > > > > Has anyone seen this issue when building latest master on Mac ? > > > > *[E

Re: gradle dirty files blocking maven build

Hi Romain, What gradle command are you running ? I don't find any ".gogradle" files. Thanks, Manu On Fri, Nov 24, 2017 at 5:09 PM Jean-Baptiste Onofré wrote: > Let me try on my local copy. > > Thanks for the report. > > Regards > JB > > On 11/24/2017 10:04 AM, Romain Manni-Bucau wrote: > > No

[Build] Enforce failed on project beam-sdks-java-io-tika

Hi all, Has anyone seen this issue when building latest master on Mac ? *[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce) on project beam-sdks-java-io-tika: Execution enforce of goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:en

Re: [VOTE] Fixing @yyy.com.INVALID mailing addresses

+1 On Thu, Nov 23, 2017 at 11:32 PM Maximilian Michels wrote: > +1 > > Thanks for looking into it! > > On 23.11.17 00:25, Lukasz Cwik wrote: > > I have noticed that some e-mail addresses (notably @google.com) get > > .INVALID suffixed onto it so per...@yyy.com become > per...@yyy.com.INVALID > >

Re: [ANNOUNCEMENT] New committers, August 2017 edition!

significant new user community. > > > > * Mingming (James) Xu > > James joined the SQL DSL effort, contributing some of the trickier parts, > > such as the Join functionality. Additionally, he's consistently shown > > himself to be an insightful code reviewer, sign

Re: [PROPOSAL] Merge gearpump-runner to master

t; Kind regards, > > Paul > > On Tue, Aug 8, 2017 at 4:10 PM, Manu Zhang > wrote: > >> Thanks Kenn!!! Thanks everyone!!! It's a great achievement for us. >> >> On Tue, Aug 8, 2017 at 7:54 AM Kenneth Knowles >> wrote: >> >> > Don

Re: [PROPOSAL] Merge gearpump-runner to master

;have > > >partial or no support for. > > > > > >I'll save practical TODOs like moving ValidatesRunner execution to > > >postcommit, etc. Pending the results of this discussion, of course. > > > > > >Kenn > > > > > > >

[PROPOSAL] Merge gearpump-runner to master

Guys, On behalf of the gearpump team, I'd like to propose to merge the gearpump-runner branch into master, which will give it more visibility to other contributors and users. The runner satisfies the following criteria outlined in contribution guide [1]. 1. Have at least 2 contributors intere

Re: Java 8?

Hi Swapnil, Try adding you module under java8 profile like java8 [1.8,) beam-sdks-java-io-geode Thanks, Manu On Sat, Jul 15, 2017 at 6:42 AM Ted Yu wrote: > Recently on Spark and Flink mailing lists there have been discussion on > d

Re: Failure in Apex runner

m/datatorrent/netlet/ folder and rebuilding, which > will fetch the netlet dependency again. > > Thanks > > On Fri, Jul 7, 2017 at 8:09 AM, Manu Zhang > wrote: > > > Hey guys, I'd like to offer some input. > > > > The test also fails locally on my Mac with th

Re: Failure in Apex runner

Hey guys, I'd like to offer some input. The test also fails locally on my Mac with the following error. (so WriteOperation#finalize is not called) java.lang.NullPointerException at com.datatorrent.netlet.util.Slice.(Slice.java:54) at org.apache.beam.runners.apex.translation.utils.ApexStateInterna

Can't run wordcount example from Intellij

Hi all, Running wordcount example from Intellij fails with "ClassNotFoundException: org.hamcrest.Matcher". This is because DirectRunner#defaultTransformOverrides will validate TestPipelineOptions, which will in turn load "hamcrest". Is this expected ? Thanks, Manu

Re: [DISCUSS] Improve error message for missing required options of Beam pipeline

Thanks, I've always been struggling with naming ;) On Wed, Jun 28, 2017 at 12:24 AM Lukasz Cwik wrote: > Thanks for the contribution, I have reviewed and merged it with minor > edits. > > On Mon, Jun 26, 2017 at 10:13 PM, Manu Zhang > wrote: > > > Sure, I&

Re: [DISCUSS] Improve error message for missing required options of Beam pipeline

a/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L182 > > On Mon, Jun 26, 2017 at 5:38 PM, Manu Zhang > wrote: > > > Thanks Kenn and Lukasz. > > > > How about we add a hidden option to `PipelineOptions` and set dependin

Re: [DISCUSS] Improve error message for missing required options of Beam pipeline

t; > code which assumed that all users were programmatic users). Should > we > > > > word > > > > > the message so its useful for both CLI and users who set the > options > > > > > programmatically? > > > > > > > > > > On Mon, Jun 26, 2017 at 6:0

Re: [DISCUSS] Improve error message for missing required options of Beam pipeline

t; JB > > On 06/25/2017 12:57 PM, Manu Zhang wrote: > > Hi all, > > > > Currently, if a required option is missing for a Beam pipeline, the error > > message is like > > > >> > >> Exception in thread "main" java.lang.IllegalAr

[DISCUSS] Improve error message for missing required options of Beam pipeline

Hi all, Currently, if a required option is missing for a Beam pipeline, the error message is like > > Exception in thread "main" java.lang.IllegalArgumentException: Missing > required value for [public abstract java.lang.String > org.apache.beam.examples.WordCount$WordCountOptions.getOutput(), "P

Re: Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4199

Looks like a network issue I've seen from time to time. On Fri, Jun 23, 2017 at 11:25 PM Jean-Baptiste Onofré wrote: > It built without problem on my machine. > > Let me check on Jenkins. > > Regards > JB > > On 06/23/2017 04:53 PM, Jean-Baptiste Onofré wrote: > > Hi Kenn, > > > > let me check,

Re: Beam Fn API

Thanks Lukasz. The following two links were somehow incorrectly formatted in your mail. * How to process a bundle: https://s.apache.org/beam-fn-api-processing-a-bundle * How to send and receive data: https://s.apache.org/beam-fn-api-send-and-receive-data By the way, is there a way to find them fr

Re: First stable release: version designation?

Slight preference for 1.0.0, but fine with 2.0.0 Thanks, Manu On Fri, May 5, 2017 at 7:24 AM Chamikara Jayalath wrote: > +1 for 2.0.0 for following reason. > > I think the main Downside for using 2.0.0 is the fact that people > incorrectly assuming this to be the second stable release. This can

Re: why SimpleDoFnRunner can not support setup

Hi, As commented here by Kenn( +k...@google.com), the name DoFnRunner is misleading and could have nothing to do with DoFn. It's used for bundle processing(startBundle, processElement, stopBundle) so there are no setup and teardown