@Yi Hu <ya...@google.com> I think adding them to Jenkins or github actions is okay with me. With Github actions, since we don't use self hosted runners yet, I worry that action workers might get queued up.
Also, I plan to not run these on every commit but run it as a cron job(maybe once per day) and also as trigger phrases and only on the lowest and highest python version. Also, migrating this workflow to jenkins would be trivial in the future once beam starts the migration. For now, I think it might be best to run on jenkins. On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev <valen...@google.com> wrote: > I think case in point dependency that would benefit from this testing is > grpcio, which includes pre-releases, and broke us and multiple of it's > released versions were yanked. https://pypi.org/project/grpcio/#history . > > We can look at how grpcio affected Beam previously. Couple of issues: > > - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests > - https://github.com/apache/beam/issues/23734 -- affected MacOS suites > - https://github.com/apache/beam/issues/22159 -- (not detected by us, but > potentially could have affected a performance test). > > I'm afraid a dedicated suite may not give us desired test coverage to > catch regression at RC stage. > > On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <dev@beam.apache.org> > wrote: > >> Thanks Anand, >> >> This would be very helpful to avoid experiencing multiple time ( >> https://s.apache.org/beam-python-dependencies-pm). One thing to note is >> that Beam Jenkins CI is experiencing many issues recently, mostly due to >> that multiple Jenkins plugins does not scale (draining GitHub API call >> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins >> if going ahead with Option 1. As we have started GitHub Action migration, >> is it considered to add these new tests to GitHub Action? >> >> Best, >> Yi >> >> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev < >> dev@beam.apache.org> wrote: >> >>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think having >>> the clear signal of the normal suite succeeding and the prerelease one >>> failing would be helpful and there shouldn't be too much additional code >>> necessary. That makes it really easy to treat the prerelease suite as a (at >>> least temporary) signal on needing upper bounds on our dependencies. >>> >>> Thanks, >>> Danny >>> >>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev < >>> dev@beam.apache.org> wrote: >>> >>>> Hi all, >>>> >>>> For Apache Beam Python we are considering using pre-released >>>> dependencies for unit testing by using the --pre flag to install >>>> pre-released dependencies of packages. >>>> >>>> We believe that using pre-released dependencies may help us to identify >>>> and resolve bugs more quickly, and to take advantage of new features or bug >>>> fixes that are not yet available in stable releases. However, we also >>>> understand that using pre-released dependencies may introduce new risks and >>>> challenges, including potential code duplication and stability issues. >>>> >>>> Before proceeding, we wanted to get your feedback on this approach. >>>> >>>> 1. Create a new PreCommit test suite and a PostCommit test suite that >>>> runs tests by installing pre-released dependencies. >>>> >>>> Pros: >>>> >>>> - stable and pre-released test suites are separate and it will be >>>> easier to debug if the pre-released test suite fails. >>>> >>>> Cons: >>>> >>>> - More test infra code to maintain. More tests to monitor. >>>> >>>> >>>> 2. Make use of the current PreCommit and PostCommit test suite and >>>> modify it so that it installs pre-released dependencies. >>>> >>>> Pros: >>>> >>>> - Less infra code and less tests to monitor. >>>> >>>> Cons: >>>> >>>> - Leads to noisy test signals if the pre-release candidate is >>>> unstable. >>>> >>>> I am in favor of approach 1 since this approach would ensure that any >>>> issues encountered during pre-release testing do not impact the stable >>>> release environment, and vice versa. >>>> >>>> If you have experience or done any testing work using pre-released >>>> dependencies, please let me know if you took any different approaches. It >>>> will be really helpful. >>>> >>>> Thanks, >>>> Anand >>>> >>>
