Thanks for the explanation, Max. We will try it out and let you guys know if we run into any questions.
Thanks, Xinyu On Thu, Apr 25, 2019 at 6:28 AM Maximilian Michels <m...@apache.org> wrote: > > - For portable running tests: by looking at the > > portableValidatesRunnerTask in flink_job_server.gradle, it seems it's > > the same set of Java tests but using portability framework to validate > > (submit to job server and run the protable pipeline in a specific > > runner). Is my understanding correct? > > That's correct. > > > - For python tests: Looks like flinkValidatesRunner is using LOOPBACK > > SDK worker type in the tests. Not sure what LOOPBACK does. Is it used > > for testing? Currently Samza portable runner supports PROCESS worker. > > This avoids building and using containers for this test. We had a number > of issues with Docker on Jenkins and wanted to lower build time for > PreCommit tests. Loopback means that an embedded Python environment will > be started which listens on localhost. It's comparable to Java's > EmbeddedSdkHarness. > > -Max > > On 24.04.19 20:10, Xinyu Liu wrote: > > Thanks for the useful pointers! We are looking forward to integrating > > both Portable and Python-specific tests for Samza runner. A few > questions: > > > > - For portable running tests: by looking at the > > portableValidatesRunnerTask in flink_job_server.gradle, it seems it's > > the same set of Java tests but using portability framework to validate > > (submit to job server and run the protable pipeline in a specific > > runner). Is my understanding correct? > > > > - For python tests: Looks like flinkValidatesRunner is using LOOPBACK > > SDK worker type in the tests. Not sure what LOOPBACK does. Is it used > > for testing? Currently Samza portable runner supports PROCESS worker. > > > > Thanks, > > Xinyu > > > > > > > > On Wed, Apr 24, 2019 at 2:45 AM Maximilian Michels <m...@apache.org > > <mailto:m...@apache.org>> wrote: > > > > > If you are interested in portable python pipeline validation, I > > think > > > fn_api_runner_test would also help. > > > > Just to note, Ankur mentioned flinkCompatibilityMatrix, that one uses > > fn_api_runner_test with some tooling on top to bring up the test > > cluster. > > > > On 23.04.19 19:23, Boyuan Zhang wrote: > > > Hi Daniel, > > > If you are interested in portable python pipeline validation, I > > think > > > fn_api_runner_test > > > > > < > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py > > > > > > > would also help. > > > > > > On Tue, Apr 23, 2019 at 10:19 AM Pablo Estrada > > <pabl...@google.com <mailto:pabl...@google.com> > > > <mailto:pabl...@google.com <mailto:pabl...@google.com>>> wrote: > > > > > > This is cool, Daniel : ) Glad to see the Samza runner moving > > forward. > > > Best > > > -P. > > > > > > On Tue, Apr 23, 2019 at 2:52 AM Maximilian Michels > > <m...@apache.org <mailto:m...@apache.org> > > > <mailto:m...@apache.org <mailto:m...@apache.org>>> wrote: > > > > > > Hi Daniel, > > > > > > Note that there is also Portable Validates Runner which > > runs Java > > > portability tests. I don't know if you have integrated > > with that > > > one > > > already. > > > > > > Thanks, > > > Max > > > > > > On 23.04.19 02:28, Ankur Goenka wrote: > > > > Hi Daniel, > > > > > > > > We use flinkCompatibilityMatrix [1] to check the Flink > > > compatibility > > > > with python. This is python equivalent to > validatesRunner > > > tests in java > > > > for portable runners. > > > > I think we can reuse it for Samza Portable runner with > > minor > > > refactoring. > > > > > > > > [1] > > > > > > > > > > https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305 > > > > > > > > On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen > > > <danx...@gmail.com <mailto:danx...@gmail.com> > > <mailto:danx...@gmail.com <mailto:danx...@gmail.com>> > > > > <mailto:danx...@gmail.com <mailto:danx...@gmail.com> > > <mailto:danx...@gmail.com <mailto:danx...@gmail.com>>>> wrote: > > > > > > > > Hi everyone, > > > > > > > > I'm working on improving the validation of the > Python > > > portable Samza > > > > runner. For java, we have the gradle task ( > > > :validatesRunner) that > > > > runs the runner validation tests. > > > > I am looking for pointers on how to similarly > > > integrate/enable the > > > > portability and Python tests for the Samza runner. > > > > > > > > Any help will be greatly appreciated. > > > > > > > > Thanks, > > > > Daniel > > > > > > > > > >
