[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=162007&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-162007
]
ASF GitHub Bot logged work on BEAM-5315:
----------------------------------------
Author: ASF GitHub Bot
Created on: 02/Nov/18 16:58
Start Date: 02/Nov/18 16:58
Worklog Time Spent: 10m
Work Description: splovyt commented on issue #6925: [BEAM-5315] Partially
port IO: avro schema parsing and codecs
URL: https://github.com/apache/beam/pull/6925#issuecomment-435444396
With these changes, the schemas finally load in correctly. Next, the avroio
test (avorio_test.py) compares the list of schemas (dictionaries) with what is
expected. The function used is the following:
(avorio_test.py line 387)
def test_read_all_from_avro_single_file(self):
path = self._write_data()
with TestPipeline() as p:
assert_that(
p \
| Create([path]) \
| avroio.ReadAllFromAvro(use_fastavro=self.use_fastavro),
equal_to(self.RECORDS))
----
the equal_to function traces back to the following implementation in
testing/util.py
(testing/util.py line 107)
# Note that equal_to always sorts the expected and actual since what we
# compare are PCollections for which there is no guaranteed order.
# However the sorting does not go beyond top level therefore [1,2] and [2,1]
# are considered equal and [[1,2]] and [[2,1]] are not.
def equal_to(expected):
expected = list(expected)
def _equal(actual):
sorted_expected = sorted(expected)
sorted_actual = sorted(actual)
if sorted_expected != sorted_actual:
raise BeamAssertException(
'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
return _equal
---
In this function, both the actual and expected list of dictionaries is
sorted before comparing. In py2 this type of sorting is possible, but not in
py3 (TypeError: unorderable types: dict() < dict()).
What would be best practice to compare two lists with dictionaries in this
custom function in Python 3? Note that the sorting does not go beyond the top
level. @tvalentyn @RobbeSneyders @Fematich @manuzhang @robertwb
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 162007)
Time Spent: 4h 40m (was: 4.5h)
> Finish Python 3 porting for io module
> -------------------------------------
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Robbe
> Assignee: Simon
> Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
