Re: Brief of interactive Beam

Hi all, I'll leave another 3 days for design <https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit?usp=sharing> review. Then we can have a vote session if there is no objection. Thanks! On Fri, Aug 9, 2019 at 12:14 PM Ning Kang wrote: > Tha

Re: Brief of interactive Beam

thogonal. > > So, in short: I think my feedback is similar to others: Can you justify > further (or reconsider) why pipeline creation and execution need to be > different? > > I can see what's the need for the watch. Can you also tell us more about > how a user would use visualize?

Re: Brief of interactive Beam

k. > > (cc'ing a few folks that might be interested +Robert Bradshaw > +Valentyn Tymofieiev +Sindy Li > +Brian Hulette ) > > Ahmet > > > On Wed, Aug 7, 2019 at 12:36 PM Ning Kang wrote: > > > To whom may concern, > > > > This is Ning from G

Re: Brief of interactive Beam

believe Robert's questions from his email has not been answered > yet. > > On Tue, Aug 13, 2019 at 5:00 PM Ning Kang wrote: > >> Hi all, I'll leave another 3 days for design >> <https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit?u

Re: Brief of interactive Beam

lt;mailto:kawai...@gmail.com>> wrote: > Thanks for the input, Robert! >> On Aug 21, 2019, at 11:49 AM, Robert Bradshaw > <mailto:rober...@google.com>> wrote: >> >> On Wed, Aug 14, 2019 at 11:29 AM Ning Kang > <mailto:ni...@google.com>> wrote: >&g

Re: Brief of interactive Beam

e replies > below. > > On Fri, Aug 23, 2019 at 1:47 PM Ning Kang <mailto:ni...@google.com>> wrote: > Thanks for the feedback, Robert! I think I got your idea. > Let me summarize it to see if it’s correct: > 1. You want everything about >> >>> standard

Re: Interactive Beam - support for caching and introspection of PCollections

nding, which add support for materializing and > accessing bounded and unbounded PCollections both from a Beam Pipeline and > from the Python interpreter. > - https://github.com/apache/beam/pull/8884 > - https://github.com/apache/beam/pull/8961 > > I am aware of the work being car

Re: portableWordCountBatch and portableWordCountStreaming failing in Python PreCommit

service On Mon, Sep 16, 2019 at 12:39 PM Ning Kang wrote: > To Ahmet, these are warnings, I'm not able to identify the errors yet. > > Thanks everyone! I'm watching the Jira now. > > On Mon, Sep 16, 2019 at 12:07 PM Chad Dombrova wrote: > >> Ning, if you're hav

portableWordCountBatch and portableWordCountStreaming failing in Python PreCommit

Hi! I've been seeing some errors during "Python PreCommit". I'm seeing "UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only. 'You are using Apache Beam with Python 2. '" Is there any plan to remove py2 tests from the pre-commit check

Re: portableWordCountBatch and portableWordCountStreaming failing in Python PreCommit

am.model.job_management.v1.JobService/GetJobMetrics is >> unimplemented","grpc_status":12}" >> > >> >> Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com >> >> >> On Mon, Sep 16, 2019 at 11:34 AM

Re: portableWordCountBatch and portableWordCountStreaming failing in Python PreCommit

>>>>>>> *jenkins-docker-apache.bintray.io/beam/flink-job-server:latest >>>>>>> <http://jenkins-docker-apache.bintray.io/beam/flink-job-server:latest>* >>>>>>> . >>>>>>> This should be introduced when we cut 2.16

Brief of interactive Beam

To whom may concern, This is Ning from Google. We are currently making efforts to leverage an interactive runner under python beam sdk. There is already an interactive Beam (iBeam for short) runner with jupyter notebook in the repo

Re: contributor permission for Beam Jira tickets

Bump this thread as a friendly ping! On Tue, Jul 16, 2019 at 5:08 PM Ning Kang wrote: > Hi, > > This is Ning Kang from Google. I'm working on the interactive beam > <https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive>. > Could

Re: [ANNOUNCE] New committer: Alan Myrvold

Congrats Alan! On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka wrote: > Congratulations Alan! > > On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang wrote: > >> Congrats, Alan! >> >> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu wrote: >> >>> Congrats, Alan! >>> >>> On Fri, Sep 27, 2019 at 10:15 AM

Re: New JIRA Component Request

Thanks, Pablo! On Tue, Oct 1, 2019 at 3:21 PM Pablo Estrada wrote: > I've created a runner-py-interactive component: > https://jira.apache.org/jira/issues/?jql=project+%3D+BEAM+AND+component+%3D+runner-py-interactive > > Hope that helps! > -P. > > On Tue, Oct 1, 2019 at 3:1

Re: New JIRA Component Request

+1 FYI, I'm temporarily using examples-python component. On Tue, Oct 1, 2019 at 3:04 PM Sam Rohde wrote: > Hi All, > > I am working improvements to the InteractiveRunner along side with +David > Yan , +Ning Kang , and Alexey > Strokach. I am requesting on behalf of this worki

Re: Confusing multiple output semantics in Python

Hi Sam, Thanks for clarifying the accessor to output when building a pipeline. Internally, we have AppliedPTransform, where the output is always a dictionary: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py#L770 And it seems to me that with key 'None', the output

Re: Python Precommit duration pushing 2 hours

ive,just separating that > out to a new suite might help. > > > > Interactive was recently added (https://github.com/apache/beam/pull/9741). > +Ning Kang could you separate interactive to new suite? > > I would ask why interactive is a separate tox configuration at all; I > don

[Interactive Beam] Changes to what to cache

*If you are not an Interactive Beam user, you can ignore this email.* Hi everyone, Recently, we've been actively developing on top of the existing InteractiveRunner for more Interactive Beam features

Re: [Interactive Beam] Changes to what to cache

Thanks Pablo! I've sent a user-geared update too. On Wed, Dec 4, 2019 at 11:16 AM Pablo Estrada wrote: > Thanks for sharing, Ning! > Is this update valuable to users as well? If so, consider sending a > user-geared update to u...@beam.apache.org. > > -P. > > On Wed, Dec 4,

Re: Python interactive runner: test dependencies removed

testenv "py37-pytest" and "py36-pytest" in tox.ini <https://github.com/apache/beam/blob/master/sdks/python/tox.ini#L100> then? @Ahmet Altay fyi On Wed, Dec 4, 2019 at 5:22 PM Pablo Estrada wrote: > +Ning Kang +Sam Rohde fyi > > On Wed, Nov 27, 2019 at 5:09 PM

Re: Python interactive runner: test dependencies removed

ironment configs. And it's invoked here: PrecommitJobBuilder builderPytest = new PrecommitJobBuilder( scope: this, nameBase: 'Python_pytest', gradleTask: ':pythonPreCommitPytest', commitTriggering: false, timeoutMins: 180, ) builderPytest.build {...} On Wed, Dec 4, 2019 at 5

Re: Python interactive runner: test dependencies removed

garding extras, go ahead in adding "interactive" to the extras option > (both py3x and py3x-pytest targets please). > > On Thu, Dec 5, 2019 at 1:55 PM Ning Kang wrote: > >> Hi Udi, >> >> Are the temporary pytest tasks in use for pre-commit check or anythi

Error logging from fn_api_runners

Hi, I just observed some error level loggings like these: ``` ERROR:apache_beam.runners.portability.fn_api_runner:created 1 workers {'worker_5': } ERROR:apache_beam.runners.portability.fn_api_runner:created 1 workers {'worker_5': } ERROR:apache_beam.runners.portability.fn_api_runner:created 1

Request edit access to Beam wiki

Hi all, I've been developing some screendiff integration tests for Interactive Beam and would like to add a few instructions in the wiki. Could I get edit access to the wiki? [image: h3rak7q86QR.png] My user ID to the confluence site is: ningk Thanks! Ning.

Re: Request edit access to Beam wiki

Thank you Pablo! I have edit access now! On Thu, Apr 9, 2020 at 10:45 AM Pablo Estrada wrote: > Hi Ning! > I've given you edit privileges for the wiki. > Thanks! > -P. > > On Thu, Apr 9, 2020 at 10:33 AM Ning Kang wrote: > >> Hi all, >> >> I've been devel

Re: [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

et Altay wrote: > >> Here you go: >> https://dist.apache.org/repos/dist/dev/beam/extensions/jupyterlab-sidepanel/v1.0.0/ >> >> On Thu, Oct 15, 2020 at 5:11 PM Robert Bradshaw >> wrote: >> >>> If we can stage the sources to dist/dev that sounds good

Re: [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

docs.google.com/document/d/1aKK8TzSrl8WiG0K4v9xZEfLMCinuGqRlMOyb7xOhgy4/edit#heading=h.he7se5yxfo7 On Fri, Oct 9, 2020 at 1:58 PM Ning Kang wrote: > To Pablo, > > The public key in use by NPM can be found in this blog "[3] > https://blog.npmjs.org/post/172999548390/new-pgp-machinery;. A di

Re: [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

t; Here you go: >> https://dist.apache.org/repos/dist/dev/beam/extensions/jupyterlab-sidepanel/v1.0.0/ >> >> On Thu, Oct 15, 2020 at 5:11 PM Robert Bradshaw >> wrote: >> >>> If we can stage the sources to dist/dev that sounds good to me. >>> >>> On

[RESULT] [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

nks, Ning and Ahmet. > > +1 (binding) Approve the release. > > On Fri, Oct 16, 2020 at 1:34 PM Ning Kang wrote: > >> Sorry, if you cannot see the missing thread history in the previous >> thread, here is another copy: >> >> On Fri, Oct 16, 2020 at 9:55 A

Re: [RESULT] [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

The package has been published to NPM: https://www.npmjs.com/package/apache-beam-jupyterlab-sidepanel Verified that JupyterLab v2+ users can use `jupyter labextension install apache-beam-jupyterlab-sidepanel` to install it now. Cheers! On Mon, Oct 19, 2020 at 11:19 AM Ning Kang wrote: >

Re: [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

You mention that NPM will sign the package. What key > will it use? We may need to upload your pgp key to the Beam list of keys? > Thanks Ning! > -P. > > On Tue, Oct 6, 2020 at 2:57 PM Ning Kang wrote: > >> Please review the release of the following jupyter labextension >> (

Re: PCollectionVisualizationTest.test_dynamic_plotting_return_handle failing in precommit

Thanks Alex! Based on the standard output, it looks like the None is returned by one of the failure paths when the environment is considered not in a notebook. There could be a race condition when tests are running in parallel and modifying the global instance ib.current_env(). In that case, a

[VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

Please review the release of the following jupyter labextension (TypeScript node package) for running Beam notebooks in JupyterLab: * apache-beam-jupyterlab-sidepanel Hi everyone, Please review and vote on the release candidate #1 for the version 1.0.0, as follows: [ ] +1, Approve the release [ ]

A whitespacelint has been added to precommit

Hi everyone, A whitespacelint precommit job has been added in this PR . The job aims to find trailing whitespaces in *non-code* files that are not scanned by each language specific linter. Now if your PR contains changes to **.md* or *build.gradle*

[VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

Hi everyone, Last week, I sent a design doc and proposals in this email thread

Interactive Beam Side Panel in JupyterLab

Hi everyone, Here is a design doc about adding a JupyterLab extension for Interactive Beam

[Question] What are the supported and recommended JDK and Python versions

Hi! Recently, I've seen questions around using incompatible JDK versions and Python versions with Beam on Stackoverflow. For example, this question about JDK and this question

Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

te: >>> >>>> +1 for [3] as well. >>>> >>>> On Wed, Jul 15, 2020 at 5:40 PM Ahmet Altay wrote: >>>> > >>>> > I agree with Kyle. [3] sounds more accurate. >>>> > >>>> > On Wed, Jul 15, 2020 a

[RESULT][VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

Hi everyone, Thanks for the vote! There are 8 votes, all of which are voting for the name `apache-beam-jupyterlab-sidepanel`: - Kyle Weaver - Ahmet Altay - Robert Bradshaw - Kenneth Knowles - Pablo Estrada - Austin Bennett - David Yan - Alexey Romanenko Thanks! Ning.

Re: Using dataflow from a notebook via Interactive Runner

Hi Sayan, 1. It's not yet officially supported to use the DataflowRunner as the underlying runner with InteractiveRunner (It's possible to set up GCS buckets with the underlying source recording and PCollection cache mechanism to work with the DataflowRunner, but it's not recommended). You can

Re: Using dataflow from a notebook via Interactive Runner

. Like how the pipeline is defined and used in the REPL environment? Are all the APIs and transforms supported? We'll have these notebooks out once those work are done. On Tue, Jan 5, 2021 at 1:55 PM Ning Kang wrote: > Hi Sayan, > > 1. It's not yet officially supported to use the DataflowRunner

Re: [VOTE] Release 2.42.0, release candidate #2

+1 Thank you, Robert! On Thu, Oct 13, 2022 at 12:47 AM Robert Burke wrote: > Hi everyone, > Please review and vote on the release candidate #2 for the version 2.42.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > >

[RESULT] [VOTE] JupyterLab Sidepanel release v3.0.0 for Issue#23488 RC#1

I'm happy to announce that we have unanimously approved this release. There are 3 approving votes, all of which are binding: * Ahmet Altay * Pablo Estrada * Chamikara Jayalath There are no disapproving votes. Thanks everyone!

Re: Is there any way to set the parallelism of operators like group by, join?

orms divided by shuffle > barrier. Would you be willing to start a follow-up thread in @dev mailing > list for this for deeper discussion? > > Jan > On 4/20/23 19:18, Ning Kang via user wrote: > > Hi Jan, > > The approach works when your pipeline doesn't have too many operator