Abacn commented on code in PR #17380:
URL: https://github.com/apache/beam/pull/17380#discussion_r860121749
##########
sdks/python/apache_beam/io/hadoopfilesystem_test.py:
##########
@@ -538,6 +539,14 @@ def test_checksum(self):
self.assertEqual(
'fake_algo-5-checksum_byte_sequence', self.fs.checksum(url))
+ def test_last_updated(self):
+ url = self.fs.join(self.tmpdir, 'f1')
+ with self.fs.create(url) as f:
+ f.write(b'Hello')
+ tolerance = 60 # 1 min
+ result = self.fs.last_updated(url)
+ self.assertAlmostEqual(result, time.time(), delta=tolerance)
Review Comment:
This mirrored an once sickbayed test
[s3io_test.test_last_updated](https://github.com/apache/beam/blob/da6acf212e93aef266630f36624f5d23a1a93801/sdks/python/apache_beam/io/aws/s3io_test.py#L116)
which set tolerance to 5 min. I did not know the reasoning of this original
setting, but once the failure has resolved, the time difference should be
instant: it is just the interval between fake file creation and the assertion
statement. If it generates confusion we could set the interval to be the same
small value in both place.
##########
sdks/python/apache_beam/io/hadoopfilesystem_test.py:
##########
@@ -36,17 +37,16 @@ class FakeFile(io.BytesIO):
"""File object for FakeHdfs"""
__hash__ = None # type: ignore[assignment]
- def __init__(self, path, mode='', type='FILE'):
+ def __init__(self, path, mode='', type='FILE', time_ms=None):
io.BytesIO.__init__(self)
-
- self.stat = {
- 'path': path,
- 'mode': mode,
- 'type': type,
- }
+ if time_ms is None:
+ time_ms = int(time.time() * 1000)
+ self.time_ms = time_ms
+ self.stat = {'path': path, 'mode': mode, 'type': type}
self.saved_data = None
def __eq__(self, other):
+ """Equality of two files. Timestamp not included in comparison"""
Review Comment:
Including timestamp would break current assertEquals used
[here](https://github.com/apache/beam/blob/7e2f746da779688657984c5987a39fb38c736b92/sdks/python/apache_beam/io/hadoopfilesystem_test.py#L368).
Considering timestamp for two "identical" file object are almost certainly
different, I did not change the behavior of `__eq__` and also the test involved
here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
