I would recommend that the known issue notice about this source at least be strongly worded - this source in the current state should be marked "DO NOT USE" - it will produce data loss in *most* production use cases. That still leaves the risk that people will use it anyway; up to folks driving the release to decide whether it's worth cutting a new candidate for the sake of just temporarily removing this source, or a notice in release notes is sufficient.
On Wed, Jul 31, 2019 at 2:59 PM Ahmet Altay <al...@google.com> wrote: > Since the python mongodb source is new in this release (not a regression) > and experimental, I agree with adding a known issues notice to the release > notes instead of starting a RC2 only for this issue. > > On Wed, Jul 31, 2019 at 2:47 PM Chamikara Jayalath <chamik...@google.com> > wrote: > >> FYI we found a critical issue with the Python MongoDB source that is >> included with this release: >> https://issues.apache.org/jira/browse/BEAM-7866 >> I suggest we include a clear notice in the release about this issue if >> the release vote has already been finalized or make this a blocker if we >> are going for a RC2. >> >> Thanks, >> Cham >> >> On Wed, Jul 31, 2019 at 2:31 AM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> On Wed, Jul 31, 2019 at 11:22 AM Valentyn Tymofieiev < >>> valen...@google.com> wrote: >>> >>>> I have checked Portable Wordcount example on Flink and Spark on Python >>>> 2 and Python 3. >>>> >>>> To do so, I had to checkout Beam from git repo, since using the source >>>> distribution does not include gradlew, and gradelw_orig did not work for >>>> me. Commands I ran: >>>> >>>> git checkout tags/v2.14.0-RC1 >>>> ./gradlew :sdks:python:container:py3:docker >>>> ./gradlew :runners:flink:1.5:job-server:runShadow # Use ./gradlew >>>> :runners:spark:job-server:runShadow for Spark >>>> ./gradlew :sdks:python:test-suites:portable:py35:portableWordCountBatch >>>> -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK >>>> cat /tmp/py-wordcount-direct* # to verify results. >>>> >>>> Loopback scenarios worked, however DOCKER scenarios did not. Opened >>>> several Jiras to follow up: >>>> >>>> https://issues.apache.org/jira/browse/BEAM-7857 >>>> https://issues.apache.org/jira/browse/BEAM-7858 >>>> https://issues.apache.org/jira/browse/BEAM-7859 >>>> <https://issues.apache.org/jira/browse/BEAM-7859?filter=-2> >>>> >>> >>> I commented on the bugs, and I think this is due to trying to use Docker >>> mode with local files (a known issue). >>> >>> >>>> The gradle targets that were required to run these tests are not >>>> present in 2.13.0 branch, so I don't consider it a regression and still >>>> cast +1. >>>> >>> >>> Agreed. >>> >>> >>>> On Wed, Jul 31, 2019 at 11:31 AM Ismaël Mejía <ieme...@gmail.com> >>>> wrote: >>>> >>>>> Oups Robert pointed to me that I have probably not counted correctly. >>>>> There were indeed already 3 PMC +1 votes. Pablo, Robert and Ahmet. >>>>> Please excuse me for the extra noise. >>>>> >>>>> On Wed, Jul 31, 2019 at 9:46 AM Ismaël Mejía <ieme...@gmail.com> >>>>> wrote: >>>>> > >>>>> > To complete the release we need to have at least three +1 binding >>>>> > votes (votes from PMC members) as stated in [1]. So far we have only >>>>> > 2. >>>>> > >>>>> > Thomas (and the others). The blog post PR is now open [2] please help >>>>> > us add missing features or maybe to highlight the ones you consider >>>>> > important in the PR comments. >>>>> > >>>>> > Here it is the missing +1 (binding). Validated SHAs+signatures, >>>>> > beam-samples and one internal company project with the new jars. >>>>> > Compared source file vs tagged git repo. Everything looks ok. >>>>> > >>>>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes >>>>> > [2] https://github.com/apache/beam/pull/9201/files >>>>> > >>>>> > On Wed, Jul 31, 2019 at 6:27 AM Anton Kedin <ke...@google.com> >>>>> wrote: >>>>> > > >>>>> > > Ran various postcommits, validates runners, and nexmark against >>>>> the release branch. All looks good so far. >>>>> > > >>>>> > > Will take another look at the docs/blog and the nexmark numbers >>>>> tomorrow, but if nothing comes up I will close the vote tomorrow >>>>> (Wednesday) by 6pm PST (= Thursday 01:00am UTC) since it's over 72hours >>>>> since the vote has started and we have a number of +1s including PMC >>>>> members and no -1s. >>>>> > > >>>>> > > Regards, >>>>> > > Anton >>>>> > > >>>>> > > On Tue, Jul 30, 2019 at 8:13 PM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> > >> >>>>> > >> I also ran unit tests for Python 3.7 and they passed as well. >>>>> Cython tests for python3.7 require `apt-get install python3.7-dev`. >>>>> > >> >>>>> > >> On Wed, Jul 31, 2019 at 3:16 AM Pablo Estrada <pabl...@google.com> >>>>> wrote: >>>>> > >>> >>>>> > >>> +1 >>>>> > >>> >>>>> > >>> I installed from source, and ran unit tests for Python in 2.7, >>>>> 3.5, 3.6. >>>>> > >>> >>>>> > >>> Also ran a number of integration tests on Py 3.5 on Dataflow and >>>>> DirectRunner. >>>>> > >>> Best >>>>> > >>> -P. >>>>> > >>> >>>>> > >>> On Tue, Jul 30, 2019 at 11:09 AM Hannah Jiang < >>>>> hannahji...@google.com> wrote: >>>>> > >>>> >>>>> > >>>> I checked Py3 tests using .zip, mainly with direct runners, and >>>>> everything looks good, so +1. >>>>> > >>>> >>>>> > >>>> On Tue, Jul 30, 2019 at 2:08 AM Robert Bradshaw < >>>>> rober...@google.com> wrote: >>>>> > >>>>> >>>>> > >>>>> I checked all the artifact signatures and ran a couple test >>>>> pipelines with the wheels (Py2 and Py3) and everything looked good to me, >>>>> so +1. >>>>> > >>>>> >>>>> > >>>>> On Mon, Jul 29, 2019 at 8:29 PM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> > >>>>>> >>>>> > >>>>>> I have checked Python 3 batch and streaming quickstarts on >>>>> Dataflow runner using .zip and wheel distributions. So far +1 from me. >>>>> > >>>>>> >>>>> > >>>>>> On Mon, Jul 29, 2019 at 7:53 PM Ahmet Altay <al...@google.com> >>>>> wrote: >>>>> > >>>>>>> >>>>> > >>>>>>> +1, validated python 2 quickstarts. >>>>> > >>>>>>> >>>>> > >>>>>>> On Fri, Jul 26, 2019 at 5:46 PM Ahmet Altay < >>>>> al...@google.com> wrote: >>>>> > >>>>>>>> >>>>> > >>>>>>>> To confirm, I manuall validated leader board on python. It >>>>> is working. >>>>> > >>>>>>>> >>>>> > >>>>>>>> On Fri, Jul 26, 2019 at 5:23 PM Yifan Zou < >>>>> yifan...@google.com> wrote: >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> AFAIK, there should not be any special prerequisites for >>>>> this. Things the script does including: >>>>> > >>>>>>>>> 1. download the python rc in zip >>>>> > >>>>>>>>> 2. start virtualenv and install the sdk. >>>>> > >>>>>>>>> 3. verify hash. >>>>> > >>>>>>>>> 4. config settings.xml and start a Java pubsub message >>>>> injector. >>>>> > >>>>>>>>> 5. run game examples and validate. >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> Could you double check if the sdk was installed properly >>>>> (step 1&2)? >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> I also guessing this is the case. Probably something >>>>> earlier in the validation script did not run as expected. >>>>> > >>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> Yifan >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> On Fri, Jul 26, 2019 at 2:38 PM Anton Kedin < >>>>> ke...@google.com> wrote: >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> Validation script fails for me when I try to run [1] >>>>> python leaderboard with direct runner: >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> ``` >>>>> > >>>>>>>>>> ***************************************************** >>>>> > >>>>>>>>>> * Running Python Leaderboard with DirectRunner >>>>> > >>>>>>>>>> ***************************************************** >>>>> > >>>>>>>>>> /usr/bin/python: No module named >>>>> apache_beam.examples.complete.game >>>>> > >>>>>>>>>> ``` >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> If someone has more context, what are the prerequisites >>>>> for this step? How does it look up the module? >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> [1] >>>>> https://github.com/apache/beam/blob/master/release/src/main/scripts/run_rc_validation.sh#L424 >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> Regards, >>>>> > >>>>>>>>>> Anton >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> On Fri, Jul 26, 2019 at 10:23 AM Anton Kedin < >>>>> ke...@google.com> wrote: >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> Cool, will make the post and will update the release >>>>> guide as well then >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> On Fri, Jul 26, 2019 at 10:20 AM Chad Dombrova < >>>>> chad...@gmail.com> wrote: >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> I think the release guide needs to be updated to >>>>> remove the optionality of blog creation and avoid confusion. Thanks for >>>>> pointing that out. >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> +1 >>>>> > >>>>>>>>>>>> >>>>> >>>>
