[
https://issues.apache.org/jira/browse/BEAM-6491?focusedWorklogId=189086&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189086
]
ASF GitHub Bot logged work on BEAM-6491:
----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Jan/19 17:34
Start Date: 23/Jan/19 17:34
Worklog Time Spent: 10m
Work Description: jklukas commented on pull request #7599: [BEAM-6491]
Remove race in FileIOTest.testMatchWatchForNewFiles
URL: https://github.com/apache/beam/pull/7599#discussion_r250297498
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java
##########
@@ -221,36 +227,33 @@ public void testMatchWatchForNewFiles() throws
IOException, InterruptedException
assertEquals(PCollection.IsBounded.UNBOUNDED, matchMetadata.isBounded());
assertEquals(PCollection.IsBounded.UNBOUNDED,
matchAllMetadata.isBounded());
+ // Copy the files to the "watch" directory, preserving the
lastModifiedTime.
Thread writer =
new Thread(
() -> {
try {
Thread.sleep(1000);
- Files.write(basePath.resolve("first"), new byte[42]);
+ Files.copy(sourcePath.resolve("first"),
watchPath.resolve("first"));
Thread.sleep(300);
- Files.write(basePath.resolve("second"), new byte[37]);
+ Files.copy(sourcePath.resolve("second"),
watchPath.resolve("second"));
Thread.sleep(300);
- Files.write(basePath.resolve("third"), new byte[99]);
+ Files.copy(sourcePath.resolve("third"),
watchPath.resolve("third"));
} catch (IOException | InterruptedException e) {
throw new RuntimeException(e);
}
});
writer.start();
+ // We fetch lastModifiedTime from the files in the "source" directory to
avoid a race condition
+ // with the writer thread.
List<MatchResult.Metadata> expected =
Arrays.asList(
metadata(
- basePath.resolve("first"),
- 42,
-
Files.getLastModifiedTime(basePath.resolve("first")).toMillis()),
+ watchPath.resolve("first"), 42,
lastModifiedMillis(sourcePath.resolve("first"))),
Review comment:
Pushed a commit to add additional commentary and to explicitly pass
`StandardCopyOption.COPY_ATTRIBUTES` to signal that we are requesting
attributes like timestamps to be preserved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 189086)
Time Spent: 2h 10m (was: 2h)
> FileIOTest.testMatchWatchForNewFiles flakey in java presubmit
> -------------------------------------------------------------
>
> Key: BEAM-6491
> URL: https://issues.apache.org/jira/browse/BEAM-6491
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution, test-failures
> Reporter: Alex Amato
> Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> I've seen this fail in a few different PRs for different contributors
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/3688/testReport/org.apache.beam.sdk.io/FileIOTest/testMatchWatchForNewFiles/]
> java.nio.file.NoSuchFileException: /tmp/junit7417716876899005708/watch/first
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> at
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.nio.file.Files.readAttributes(Files.java:1737) at
> java.nio.file.Files.getLastModifiedTime(Files.java:2266) at
> org.apache.beam.sdk.io.FileIOTest.testMatchWatchForNewFiles(FileIOTest.java:243)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319) at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:265)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:349) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:314) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:312) at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:396) at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
> at
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
> at java.lang.Thread.run(Thread.java:748)
> h3. Standard Error
> [direct-runner-worker] INFO org.apache.beam.sdk.io.FileIO - Matched 1 files
> for pattern /tmp/junit2386911851172406517/file [direct-runner-worker] INFO
> org.apache.beam.sdk.io.FileIO - Matched 1 files for pattern
> /tmp/junit2386911851172406517/file.gz Exception in thread "Thread-4"
> java.lang.RuntimeException: java.nio.file.NoSuchFileException:
> /tmp/junit7417716876899005708/watch/first at
> org.apache.beam.sdk.io.FileIOTest.lambda$testMatchWatchForNewFiles$0(FileIOTest.java:233)
> at java.lang.Thread.run(Thread.java:748) Caused by:
> java.nio.file.NoSuchFileException: /tmp/junit7417716876899005708/watch/first
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at
> java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
> at java.nio.file.Files.newOutputStream(Files.java:216) at
> java.nio.file.Files.write(Files.java:3292) at
> org.apache.beam.sdk.io.FileIOTest.lambda$testMatchWatchForNewFiles$0(FileIOTest.java:227)
> ... 1 more [
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
