The issue is only visible with Python 3.6, not 2.7. If there is a framework in place to add a streaming test, that would be great. We would use what we have internally as starting point.
On Thu, Sep 5, 2019 at 5:00 PM Ahmet Altay <al...@google.com> wrote: > > > On Thu, Sep 5, 2019 at 4:15 PM Thomas Weise <t...@apache.org> wrote: > >> The workload is quite different. What I have is streaming with state and >> timers. >> >> >> >> On Thu, Sep 5, 2019 at 3:47 PM Pablo Estrada <pabl...@google.com> wrote: >> >>> We only recently started running Chicago Taxi Example. +Michał Walenia >>> <michal.wale...@polidea.com> I don't see it in the dashboards. Do you >>> know if it's possible to see any trends in the data? >>> >>> We have a few tests running now: >>> - Combine tests: >>> https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792&widget=201943890&container=1334074373 >>> - GBK tests: >>> https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792&widget=201943890&container=1334074373 >>> >>> They don't seem to show a very drastic jump either, but they aren't very >>> old. >>> >>> There is also work ongoing to add alerting for this sort of regressions >>> by Kasia and Kamil (added). The work is not there yet (it's in progress). >>> Best >>> -P. >>> >>> On Thu, Sep 5, 2019 at 3:35 PM Thomas Weise <t...@apache.org> wrote: >>> >>>> It probably won't be practical to do a bisect due to the high cost of >>>> each iteration with our fork/deploy setup. >>>> >>>> Perhaps it is time to setup something with the synthetic source that >>>> works just with Beam as dependency. >>>> >>> > I agree with this. > > Pablo, Kasia, Kamil, does the new benchmarks give us a easy to use > framework for using synthetic source in benchmarks? > > >> >>>> On Thu, Sep 5, 2019 at 3:23 PM Ahmet Altay <al...@google.com> wrote: >>>> >>>>> There are a few in this dashboard [1], but not very useful in this >>>>> case because they do not go back more than a month and not very >>>>> comprehensive. I do not see a jump there. Thomas, would it be possible to >>>>> bisect to find what commit caused the regression? >>>>> >>>>> +Pablo Estrada <pabl...@google.com> do we have any python on flink >>>>> benchmarks for chicago example? >>>>> +Alan Myrvold <amyrv...@google.com> +Yifan Zou <yifan...@google.com> It >>>>> would be good to have alerts on benchmarks. Do we have such an ability >>>>> today? >>>>> >>>>> [1] https://apache-beam-testing.appspot.com/dashboard-admin >>>>> >>>>> On Thu, Sep 5, 2019 at 3:15 PM Thomas Weise <t...@apache.org> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Are there any performance tests run for the Python SDK as part of >>>>>> release verification (or otherwise as well)? >>>>>> >>>>>> I see what appears to be a regression in master (compared to 2.14) >>>>>> with our in-house application (~ 25% jump in cpu utilization and >>>>>> corresponds drop in throughput). >>>>>> >>>>>> I wanted to see if there is anything available to verify that within >>>>>> Beam. >>>>>> >>>>>> Thanks, >>>>>> Thomas >>>>>> >>>>>>
