+Valentyn Tymofieiev <valen...@google.com> do we have benchmarks in different python versions? Was there a recent change that is specific to python 3.x ?
On Fri, Sep 6, 2019 at 8:36 AM Thomas Weise <t...@apache.org> wrote: > The issue is only visible with Python 3.6, not 2.7. > > If there is a framework in place to add a streaming test, that would be > great. We would use what we have internally as starting point. > > On Thu, Sep 5, 2019 at 5:00 PM Ahmet Altay <al...@google.com> wrote: > >> >> >> On Thu, Sep 5, 2019 at 4:15 PM Thomas Weise <t...@apache.org> wrote: >> >>> The workload is quite different. What I have is streaming with state and >>> timers. >>> >>> >>> >>> On Thu, Sep 5, 2019 at 3:47 PM Pablo Estrada <pabl...@google.com> wrote: >>> >>>> We only recently started running Chicago Taxi Example. +Michał Walenia >>>> <michal.wale...@polidea.com> I don't see it in the dashboards. Do you >>>> know if it's possible to see any trends in the data? >>>> >>>> We have a few tests running now: >>>> - Combine tests: >>>> https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792&widget=201943890&container=1334074373 >>>> - GBK tests: >>>> https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792&widget=201943890&container=1334074373 >>>> >>>> They don't seem to show a very drastic jump either, but they aren't >>>> very old. >>>> >>>> There is also work ongoing to add alerting for this sort of regressions >>>> by Kasia and Kamil (added). The work is not there yet (it's in progress). >>>> Best >>>> -P. >>>> >>>> On Thu, Sep 5, 2019 at 3:35 PM Thomas Weise <t...@apache.org> wrote: >>>> >>>>> It probably won't be practical to do a bisect due to the high cost of >>>>> each iteration with our fork/deploy setup. >>>>> >>>>> Perhaps it is time to setup something with the synthetic source that >>>>> works just with Beam as dependency. >>>>> >>>> >> I agree with this. >> >> Pablo, Kasia, Kamil, does the new benchmarks give us a easy to use >> framework for using synthetic source in benchmarks? >> >> >>> >>>>> On Thu, Sep 5, 2019 at 3:23 PM Ahmet Altay <al...@google.com> wrote: >>>>> >>>>>> There are a few in this dashboard [1], but not very useful in this >>>>>> case because they do not go back more than a month and not very >>>>>> comprehensive. I do not see a jump there. Thomas, would it be possible to >>>>>> bisect to find what commit caused the regression? >>>>>> >>>>>> +Pablo Estrada <pabl...@google.com> do we have any python on flink >>>>>> benchmarks for chicago example? >>>>>> +Alan Myrvold <amyrv...@google.com> +Yifan Zou <yifan...@google.com> It >>>>>> would be good to have alerts on benchmarks. Do we have such an ability >>>>>> today? >>>>>> >>>>>> [1] https://apache-beam-testing.appspot.com/dashboard-admin >>>>>> >>>>>> On Thu, Sep 5, 2019 at 3:15 PM Thomas Weise <t...@apache.org> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Are there any performance tests run for the Python SDK as part of >>>>>>> release verification (or otherwise as well)? >>>>>>> >>>>>>> I see what appears to be a regression in master (compared to 2.14) >>>>>>> with our in-house application (~ 25% jump in cpu utilization and >>>>>>> corresponds drop in throughput). >>>>>>> >>>>>>> I wanted to see if there is anything available to verify that within >>>>>>> Beam. >>>>>>> >>>>>>> Thanks, >>>>>>> Thomas >>>>>>> >>>>>>>
