[jira] [Commented] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036345#comment-17036345 ] Valentyn Tymofieiev commented on BEAM-7455: --- With #9466, Py3 integration test coverage of Avro IO matches Py2 test coverage. This is done via fastavro_it_test.py, which runs a pipeline using Avro and FastAvro. If/we drop avro dependency, we may need to update this test. > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-py-avro >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907167#comment-16907167 ] Valentyn Tymofieiev commented on BEAM-7455: --- There is a PR in flight, taking this over as an action to review & merge. > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-py-avro >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853485#comment-16853485 ] Chamikara Jayalath commented on BEAM-7455: -- I think it's good if we can add an integration test as well. For example, Java has an AvroIO performance test here which we could replicate for Python: [https://builds.apache.org/view/A-D/view/Beam/view/PerformanceTests/job/beam_PerformanceTests_AvroIOIT/] > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851709#comment-16851709
]
Frederik Bode commented on BEAM-7455:
-
Right now the IT test compares files that are read and written with the avro
packages to files that are read and written with the fastavro package. If we
cannot compare the two, what should we test? Testing whether the packages work
individually is already done in \{{ avroio_test.py }}.
> Improve Avro IO integration test coverage on Python 3.
> --
>
> Key: BEAM-7455
> URL: https://issues.apache.org/jira/browse/BEAM-7455
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Valentyn Tymofieiev
>Assignee: Frederik Bode
>Priority: Major
>
> It seems that we don't have an integration test for Avro IO on Python 3:
> fastavro_it_test [1] depends on both avro and fastavro, however avro package
> currently does not work with Beam on Python 3, so we don't have an
> integration test that exercises Avro IO on Python 3.
> We should add an integration test for Avro IO that does not need both
> libraries at the same time, and instead can run using either library.
> [~frederik] is this something you could help with?
> cc: [~chamikara] [~Juta]
> [1]
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
