[
https://issues.apache.org/jira/browse/BEAM-6748?focusedWorklogId=211084&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-211084
]
ASF GitHub Bot logged work on BEAM-6748:
----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Mar/19 15:48
Start Date: 11/Mar/19 15:48
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8015: [BEAM-6748]
Account for synchronization interval when estimating amount of blocks in
generated Avro test file.
URL: https://github.com/apache/beam/pull/8015#discussion_r264298186
##########
File path: sdks/python/apache_beam/io/avroio_test.py
##########
@@ -299,17 +298,21 @@ def test_split_points(self):
for _ in splits[0].source.read(range_tracker):
split_points_report.append(range_tracker.split_points())
- # There are a total of three blocks. Each block has more than 10 records.
+ # There will be a total of num_blocks in the generated test file,
+ # proportional to number of records in the file divided by syncronization
+ # interval used by avro during write. Each block has more than 10 records.
+ num_blocks = int(math.ceil(14.5 * num_records /
+ avro.datafile.SYNC_INTERVAL))
# When reading records of the first block, range_tracker.split_points()
# should return (0, iobase.RangeTracker.SPLIT_POINTS_UNKNOWN)
- self.assertEquals(
+ self.assertEqual(
Review comment:
Done, thanks you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 211084)
Time Spent: 40m (was: 0.5h)
> Block size difference in avro library on Python3 causes some AvroIO tests to
> fail.
> ----------------------------------------------------------------------------------
>
> Key: BEAM-6748
> URL: https://issues.apache.org/jira/browse/BEAM-6748
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *apache_beam.io.avroio_test.TestAvro.test_split_points*
> *apache_beam.io.avroio_test.TestFastAvro.test_split_points*
> fail with:
>
> {code:java}
> Traceback (most recent call last):
> File "/home/robbe/workspace/beam/sdks/python/apache_beam/io/avroio_test.py",
> line 308, in test_split_points
> self.assertEquals(split_points_report[-10:], [(2, 1)] * 10)
> AssertionError: Lists differ: [(10, 1), (10, 1), (10, 1), (10, 1), (10, 1[42
> chars], 1)] != [(2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2[32 chars], 1)]
> First differing element 0:
> (10, 1)
> (2, 1)
> + [(2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2, 1),
> (2, 1)]
> - [(10, 1),
> - (10, 1),
> - (10, 1),
> - (10, 1),
> - (10, 1),
> - (10, 1),
> - (10, 1),
> - (10, 1),
> - (10, 1),
> - (10, 1)] {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
