[jira] [Commented] (BEAM-7365) apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow
[
https://issues.apache.org/jira/browse/BEAM-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845504#comment-16845504
]
Valentyn Tymofieiev commented on BEAM-7365:
---
The problem is with the test. This is not a release blocker.
https://github.com/apache/beam/pull/8646 out to fix the rootcause.
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> is very slow
> -
>
> Key: BEAM-7365
> URL: https://issues.apache.org/jira/browse/BEAM-7365
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Robert Bradshaw
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestFastAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestFastAvro) ... WARNING:root:After 101
> concurrent splitting trials at item #2, observed only failure, giving up on
> this item
> WARNING:root:After 101 concurrent splitting trials at item #21, observed only
> failure, giving up on this item
> WARNING:root:After 101 concurrent splitting trials at item #22, observed only
> failure, giving up on this item
> WARNING:root:After 1014 total concurrent splitting trials, considered only 25
> items, giving up.
> ok
> --
> Ran 1 test in 172.223s
>
> {noformat}
> Compare this with
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestAvro) ... ok
> --
> Ran 1 test in 0.623s
> OK
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7365) apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow
[
https://issues.apache.org/jira/browse/BEAM-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845192#comment-16845192
]
Valentyn Tymofieiev commented on BEAM-7365:
---
[~angoenka] I would prefer to fix this issue on the release branch, since this
may be a regression for FastAvro users in 2.13.0, I'll see if we can do a fix,
or if not will consider a rollback of
https://github.com/apache/beam/commit/0ac265b59a679804d105cc05c8a17f0c31402030
for the release branch.
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> is very slow
> -
>
> Key: BEAM-7365
> URL: https://issues.apache.org/jira/browse/BEAM-7365
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Robert Bradshaw
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.13.0
>
>
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestFastAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestFastAvro) ... WARNING:root:After 101
> concurrent splitting trials at item #2, observed only failure, giving up on
> this item
> WARNING:root:After 101 concurrent splitting trials at item #21, observed only
> failure, giving up on this item
> WARNING:root:After 101 concurrent splitting trials at item #22, observed only
> failure, giving up on this item
> WARNING:root:After 1014 total concurrent splitting trials, considered only 25
> items, giving up.
> ok
> --
> Ran 1 test in 172.223s
>
> {noformat}
> Compare this with
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestAvro) ... ok
> --
> Ran 1 test in 0.623s
> OK
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7365) apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow
[
https://issues.apache.org/jira/browse/BEAM-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845138#comment-16845138
]
Valentyn Tymofieiev commented on BEAM-7365:
---
I also verified that the slowness appeared after
https://github.com/apache/beam/commit/0ac265b59a679804d105cc05c8a17f0c31402030.
If we check out a parent commit, the test passes:
git checkout c25b34a569b979301337a76c5a92e0f438555f78
test_dynamic_work_rebalancing_exhaustive
(apache_beam.io.avroio_test.TestFastAvro) ... ok
python$ python setup.py test -s
apache_beam.io.avroio_test.TestFastAvro.test_dynamic_work_rebalancing_exhaustive
--
Ran 1 test in 0.132s
OK
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> is very slow
> -
>
> Key: BEAM-7365
> URL: https://issues.apache.org/jira/browse/BEAM-7365
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Robert Bradshaw
>Assignee: Frederik Bode
>Priority: Major
>
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestFastAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestFastAvro) ... WARNING:root:After 101
> concurrent splitting trials at item #2, observed only failure, giving up on
> this item
> WARNING:root:After 101 concurrent splitting trials at item #21, observed only
> failure, giving up on this item
> WARNING:root:After 101 concurrent splitting trials at item #22, observed only
> failure, giving up on this item
> WARNING:root:After 1014 total concurrent splitting trials, considered only 25
> items, giving up.
> ok
> --
> Ran 1 test in 172.223s
>
> {noformat}
> Compare this with
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestAvro) ... ok
> --
> Ran 1 test in 0.623s
> OK
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7365) apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow
[
https://issues.apache.org/jira/browse/BEAM-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845130#comment-16845130
]
Valentyn Tymofieiev commented on BEAM-7365:
---
It is also very slow on Python 2, so the slowness is not related to changes of
threading model on Python 3.
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> is very slow
> -
>
> Key: BEAM-7365
> URL: https://issues.apache.org/jira/browse/BEAM-7365
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Robert Bradshaw
>Assignee: Frederik Bode
>Priority: Major
>
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestFastAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestFastAvro) ... WARNING:root:After 101
> concurrent splitting trials at item #2, observed only failure, giving up on
> this item
> WARNING:root:After 101 concurrent splitting trials at item #21, observed only
> failure, giving up on this item
> WARNING:root:After 101 concurrent splitting trials at item #22, observed only
> failure, giving up on this item
> WARNING:root:After 1014 total concurrent splitting trials, considered only 25
> items, giving up.
> ok
> --
> Ran 1 test in 172.223s
>
> {noformat}
> Compare this with
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestAvro) ... ok
> --
> Ran 1 test in 0.623s
> OK
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7365) apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow
[
https://issues.apache.org/jira/browse/BEAM-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845127#comment-16845127
]
Valentyn Tymofieiev commented on BEAM-7365:
---
Original description had a typo. Avro test is fine, but FastAvro test is very
slow. Repro command:
python setup.py test -s
apache_beam.io.avroio_test.TestFastAvro.test_dynamic_work_rebalancing_exhaustive
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> is very slow
> -
>
> Key: BEAM-7365
> URL: https://issues.apache.org/jira/browse/BEAM-7365
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Robert Bradshaw
>Assignee: Frederik Bode
>Priority: Major
>
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestFastAvro) ... WARNING:root:After 101
> concurrent splitting trials at item #2, observed only failure, giving up on
> this item
> WARNING:root:After 101 concurrent splitting trials at item #21, observed only
> failure, giving up on this item
> WARNING:root:After 101 concurrent splitting trials at item #22, observed only
> failure, giving up on this item
> WARNING:root:After 1014 total concurrent splitting trials, considered only 25
> items, giving up.
> ok
> --
> Ran 1 test in 172.223s
>
> {noformat}
> Compare this with
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestAvro) ... ok
> --
> Ran 1 test in 0.623s
> OK
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7365) apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow
[
https://issues.apache.org/jira/browse/BEAM-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844689#comment-16844689
]
Frederik Bode commented on BEAM-7365:
-
I cannot reproduce this behaviour:
Python 2.7:
```
py27 inst-nodeps:
/app/sdks/python/target/.tox/.tmp/package/1/apache-beam-2.14.0.dev0.zip
...
py27 run-test: commands[0] | python setup.py test -s
apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
...
test_dynamic_work_rebalancing_exhaustive (apache_beam.io.avroio_test.TestAvro)
... WARNING:root:After 101 concurrent splitting trials at item #3, observed
only failure, giving up on this item
ok
--
Ran 1 test in 0.705s
OK
py27 run-test-post: commands[0] | /app/sdks/python/scripts/run_tox_cleanup.sh
...
```
Python 3.5
```
py35 inst-nodeps:
/app/sdks/python/target/.tox/.tmp/package/1/apache-beam-2.14.0.dev0.zip
...
py35 run-test: commands[0] | python setup.py test -s
apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
...
test_dynamic_work_rebalancing_exhaustive (apache_beam.io.avroio_test.TestAvro)
... /app/sdks/python/apache_beam/io/source_test_utils.py:596:
DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
'success' if have_success else 'failure'
WARNING:root:After 101 concurrent splitting trials at item #1, observed only
failure, giving up on this item
WARNING:root:After 101 concurrent splitting trials at item #2, observed only
failure, giving up on this item
WARNING:root:After 101 concurrent splitting trials at item #3, observed only
failure, giving up on this item
WARNING:root:After 101 concurrent splitting trials at item #4, observed only
failure, giving up on this item
ok
--
Ran 1 test in 1.201s
OK
```
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> is very slow
> -
>
> Key: BEAM-7365
> URL: https://issues.apache.org/jira/browse/BEAM-7365
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Robert Bradshaw
>Priority: Major
>
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestFastAvro) ... WARNING:root:After 101
> concurrent splitting trials at item #2, observed only failure, giving up on
> this item
> WARNING:root:After 101 concurrent splitting trials at item #21, observed only
> failure, giving up on this item
> WARNING:root:After 101 concurrent splitting trials at item #22, observed only
> failure, giving up on this item
> WARNING:root:After 1014 total concurrent splitting trials, considered only 25
> items, giving up.
> ok
> --
> Ran 1 test in 172.223s
>
> {noformat}
> Compare this with
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestAvro) ... ok
> --
> Ran 1 test in 0.623s
> OK
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7365) apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow
[
https://issues.apache.org/jira/browse/BEAM-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844145#comment-16844145
]
Chamikara Jayalath commented on BEAM-7365:
--
Talking a look. Seems like test works fine for Py2. So probably have to be
redesigned to fit the threading model of Py3.
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> is very slow
> -
>
> Key: BEAM-7365
> URL: https://issues.apache.org/jira/browse/BEAM-7365
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Robert Bradshaw
>Priority: Major
>
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestFastAvro) ... WARNING:root:After 101
> concurrent splitting trials at item #2, observed only failure, giving up on
> this item
> WARNING:root:After 101 concurrent splitting trials at item #21, observed only
> failure, giving up on this item
> WARNING:root:After 101 concurrent splitting trials at item #22, observed only
> failure, giving up on this item
> WARNING:root:After 1014 total concurrent splitting trials, considered only 25
> items, giving up.
> ok
> --
> Ran 1 test in 172.223s
>
> {noformat}
> Compare this with
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestAvro) ... ok
> --
> Ran 1 test in 0.623s
> OK
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7365) apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive is very slow
[
https://issues.apache.org/jira/browse/BEAM-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843860#comment-16843860
]
Robert Bradshaw commented on BEAM-7365:
---
I am concerned that this indicates less opportunities for splitting as well,
given the warnings.
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> is very slow
> -
>
> Key: BEAM-7365
> URL: https://issues.apache.org/jira/browse/BEAM-7365
> Project: Beam
> Issue Type: New Feature
> Components: io-python-avro
>Reporter: Robert Bradshaw
>Priority: Major
>
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestFastAvro) ... WARNING:root:After 101
> concurrent splitting trials at item #2, observed only failure, giving up on
> this item
> WARNING:root:After 101 concurrent splitting trials at item #21, observed only
> failure, giving up on this item
> WARNING:root:After 101 concurrent splitting trials at item #22, observed only
> failure, giving up on this item
> WARNING:root:After 1014 total concurrent splitting trials, considered only 25
> items, giving up.
> ok
> --
> Ran 1 test in 172.223s
>
> {noformat}
> Compare this with
> {noformat}
> $ python setup.py test -s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive
> test_dynamic_work_rebalancing_exhaustive
> (apache_beam.io.avroio_test.TestAvro) ... ok
> --
> Ran 1 test in 0.623s
> OK
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
