[
https://issues.apache.org/jira/browse/BEAM-6491?focusedWorklogId=189074&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189074
]
ASF GitHub Bot logged work on BEAM-6491:
----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Jan/19 17:20
Start Date: 23/Jan/19 17:20
Worklog Time Spent: 10m
Work Description: ajamato commented on pull request #7599: [BEAM-6491]
Remove race in FileIOTest.testMatchWatchForNewFiles
URL: https://github.com/apache/beam/pull/7599#discussion_r250291901
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java
##########
@@ -221,36 +227,33 @@ public void testMatchWatchForNewFiles() throws
IOException, InterruptedException
assertEquals(PCollection.IsBounded.UNBOUNDED, matchMetadata.isBounded());
assertEquals(PCollection.IsBounded.UNBOUNDED,
matchAllMetadata.isBounded());
+ // Copy the files to the "watch" directory, preserving the
lastModifiedTime.
Thread writer =
new Thread(
() -> {
try {
Thread.sleep(1000);
- Files.write(basePath.resolve("first"), new byte[42]);
+ Files.copy(sourcePath.resolve("first"),
watchPath.resolve("first"));
Thread.sleep(300);
- Files.write(basePath.resolve("second"), new byte[37]);
+ Files.copy(sourcePath.resolve("second"),
watchPath.resolve("second"));
Thread.sleep(300);
- Files.write(basePath.resolve("third"), new byte[99]);
+ Files.copy(sourcePath.resolve("third"),
watchPath.resolve("third"));
} catch (IOException | InterruptedException e) {
throw new RuntimeException(e);
}
});
writer.start();
+ // We fetch lastModifiedTime from the files in the "source" directory to
avoid a race condition
+ // with the writer thread.
List<MatchResult.Metadata> expected =
Arrays.asList(
metadata(
- basePath.resolve("first"),
- 42,
-
Files.getLastModifiedTime(basePath.resolve("first")).toMillis()),
+ watchPath.resolve("first"), 42,
lastModifiedMillis(sourcePath.resolve("first"))),
Review comment:
Is the lastModifiedTime changed when the sourcePath files are written? Why
do we expect this match when the file are written back out a second time in the
watchPath dir?
Or is the last modified time copied from the sourcePath files? So we expect
it to be the same?
Would you please add a comment in this test clarifying this for the next
reader?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 189074)
Time Spent: 2h (was: 1h 50m)
> FileIOTest.testMatchWatchForNewFiles flakey in java presubmit
> -------------------------------------------------------------
>
> Key: BEAM-6491
> URL: https://issues.apache.org/jira/browse/BEAM-6491
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution, test-failures
> Reporter: Alex Amato
> Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> I've seen this fail in a few different PRs for different contributors
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/3688/testReport/org.apache.beam.sdk.io/FileIOTest/testMatchWatchForNewFiles/]
> java.nio.file.NoSuchFileException: /tmp/junit7417716876899005708/watch/first
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> at
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.nio.file.Files.readAttributes(Files.java:1737) at
> java.nio.file.Files.getLastModifiedTime(Files.java:2266) at
> org.apache.beam.sdk.io.FileIOTest.testMatchWatchForNewFiles(FileIOTest.java:243)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319) at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:265)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:349) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:314) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:312) at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:396) at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
> at
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
> at java.lang.Thread.run(Thread.java:748)
> h3. Standard Error
> [direct-runner-worker] INFO org.apache.beam.sdk.io.FileIO - Matched 1 files
> for pattern /tmp/junit2386911851172406517/file [direct-runner-worker] INFO
> org.apache.beam.sdk.io.FileIO - Matched 1 files for pattern
> /tmp/junit2386911851172406517/file.gz Exception in thread "Thread-4"
> java.lang.RuntimeException: java.nio.file.NoSuchFileException:
> /tmp/junit7417716876899005708/watch/first at
> org.apache.beam.sdk.io.FileIOTest.lambda$testMatchWatchForNewFiles$0(FileIOTest.java:233)
> at java.lang.Thread.run(Thread.java:748) Caused by:
> java.nio.file.NoSuchFileException: /tmp/junit7417716876899005708/watch/first
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at
> java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
> at java.nio.file.Files.newOutputStream(Files.java:216) at
> java.nio.file.Files.write(Files.java:3292) at
> org.apache.beam.sdk.io.FileIOTest.lambda$testMatchWatchForNewFiles$0(FileIOTest.java:227)
> ... 1 more [
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
