[jira] [Resolved] (COMPRESS-591) Fix decoding of 7z files containing LZMA streams with end marker
[
https://issues.apache.org/jira/browse/COMPRESS-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Lee resolved COMPRESS-591.
Fix Version/s: 1.22
Resolution: Fixed
> Fix decoding of 7z files containing LZMA streams with end marker
>
>
> Key: COMPRESS-591
> URL: https://issues.apache.org/jira/browse/COMPRESS-591
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Lasse Collin
>Priority: Major
> Fix For: 1.22
>
> Attachments: lzma-with-eos.7z
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Some uncommon but valid .7z files contain LZMA streams that use the end of
> stream marker. Currently Commons Compress together with XZ for Java considers
> such files to be corrupt.
> XZ for Java 1.9 added a new method
> [LZMAInputStream.enableRelaxedEndCondition()|https://tukaani.org/xz/xz-javadoc/org/tukaani/xz/LZMAInputStream.html#enableRelaxedEndCondition()]
> specifically for this issue. To use this feature in Commons Compress, a
> change is needed to the "decode" function in
> src/main/java/org/apache/commons/compress/archivers/sevenz/LZMADecoder.java:
> {{- return new LZMAInputStream(in, uncompressedLength, propsByte, dictSize);}}
> {{+ final LZMAInputStream lzmaIn = new LZMAInputStream(in,
> uncompressedLength, propsByte, dictSize);}}
> {{+ lzmaIn.enableRelaxedEndCondition();}}
> {{+ return lzmaIn;}}
> A tiny test file is attached (thanks to Simon for providing it). Another test
> file "sheet.7z" can be found from
> <[https://sourceforge.net/p/lzmautils/discussion/708858/thread/822d80d5ea/]>.
> XZ for Java 1.9 is already a few months old, so I apologize for not reporting
> this earlier.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (COMPRESS-592) Checksum verification failed reading 7z archive with more than 65536 entries
[
https://issues.apache.org/jira/browse/COMPRESS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Lee resolved COMPRESS-592.
Fix Version/s: 1.22
Resolution: Fixed
> Checksum verification failed reading 7z archive with more than 65536 entries
>
>
> Key: COMPRESS-592
> URL: https://issues.apache.org/jira/browse/COMPRESS-592
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.21
> Environment: Compress 1.21 and XZ 1.9 on JDK 11; reproduced on both
> Windows and Ubuntu Linux
>Reporter: Roland Kreuzer
>Priority: Major
> Fix For: 1.22
>
> Attachments: _DOC.7z
>
>
> I have a use-case where I have to decompress Sevenzip archives from an
> external source which may have a large number of entries.
> I found decompression fails when trying to extract entry 65536 (zero-based
> index) with a checksum failure.
>
> I was able to reproduce the issue with a simple 7Zip file containing 70.001
> entries with random MD5 checksum textfiles (attached).
> The sample Archive was created using the 7Zip Windows client and uses
> LZMA2:3m.
>
> My code is a simple sequential read of all contents of the file like
> {code:java}
> @Test
> void readBigSevenZipFile() throws IOException
> {
> try (SevenZFile sevenZFile = new SevenZFile(new
> File("E:\\Temp\\_DOC.7z")))
> {
> SevenZArchiveEntry entry = sevenZFile.getNextEntry();
> while (entry != null)
> {
> if (entry.hasStream())
> {
> byte[] content = new byte[(int) entry.getSize()];
> sevenZFile.read(content);
> System.out.println(entry.getName());
> }
> entry = sevenZFile.getNextEntry();
> }
> }
> }
> {code}
> which fails consistently after file65535.txt with
> {code:java}
> java.io.IOException: Checksum verification failed
> at
> org.apache.commons.compress.utils.ChecksumVerifyingInputStream.read(ChecksumVerifyingInputStream.java:94)
> ~[commons-compress-1.21.jar!/:1.21]
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.read(SevenZFile.java:1905)
> ~[commons-compress-1.21.jar!/:1.21]
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.read(SevenZFile.java:1888)
> ~[commons-compress-1.21.jar!/:1.21]
> {code}
>
> It is noticeable that the value is 2 to the 16th power, which could suggest
> an overflow error of some sorts.
>
> While the minimal sample contains only small txt files, I originally found
> the issue with larger archives containing also Image and PDF files. The
> archive's contents or size in byte does not seem to have direct influence on
> the issue, only the number of files contained within.
>
> I did not find any workaround yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436000#comment-17436000
]
Gary D. Gregory edited comment on IO-755 at 10/29/21, 6:11 PM:
---
How about using {{PathUtils.walk(Path, PathFilter, int, boolean,
FileVisitOption...)}} and passing {{false}} to {{readAttributes}} which is what
{{FileUtils.streamFiles(File, boolean, String...)}} and consequently
{{FileUtils.iterateFiles(File, String[], boolean)}} and
{{FileUtils.listFiles(File, String[], boolean)}} do.
It would be nice if {{java.nio.file.FileVisitOption}} had a
{{DONT_READ_FILE_ATTRIBUTES}} value I suppose as it looks like attribute
reading in baked-in file walking.
was (Author: garydgregory):
How about using {{PathUtils.walk(Path, PathFilter, int, boolean,
FileVisitOption...)}} and passing {{false}} to {{readAttributes}} which is what
{{FileUtils.streamFiles(File, boolean, String...)}} and consequently
{{FileUtils.iterateFiles(File, String[], boolean)}} and
{{FileUtils.listFiles(File, String[], boolean)}} do.
It would be nice if {{java.nio.file.FileVisitOption}} had a
{{DONT_READ_FILE_ATTRIBUTES}} value I suppose.
> Using FileUtils.listFiles with background changes fails on Linux
>
>
> Key: IO-755
> URL: https://issues.apache.org/jira/browse/IO-755
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Alexis Suter
>Priority: Major
> Labels: listFiles
>
> When calling {{FileUtils.listFiles}} and a file is being removed from this or
> a subdirectory during the execution the following Error occurs on linux.
> h3. Stacktrace
> {code:trace}
> Caused by: java.nio.file.NoSuchFileException: /**
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
> at
> java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
> at
> java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
> at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
> at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
> at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
> at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
> at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
> {code}
> h3. Upgrade of apache commons-io
> The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the
> library started using java.nio instead of java.io which might behave
> differently on unix, when a file is not existent during processing of
> directories.
> h3. Causing functionality in our implementation
> In some part we have multithreaded access to the filesystem. A folder is
> accessed from multiple threads which access the folder with
> {{org.apache.commons.io.FileUtils#listFiles(java.io.File,
> org.apache.commons.io.filefilter.IOFileFilter,
> org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
> *delete* and *rename* files from this directory. While this seems to be
> working on a windows machine, it is not the case on a Linux machine. In 2.8
> apparently there was no issue with this.
> *In short*: While a directory is being listed with {{FileUtils.listFiles()}},
> a file is being deleted. When trying to read its file-attributes it stops
> because of a {{NoSuchFileException}}.
> h3. Other commons-io functionalities
> We tried using {{PathUtils.walk(directory filters, 5,
> false).map(Path::toFile).collect(Collectors.toList())}}. This method was
> introduced with the same change.
> This produced a similar stacktrace:
> {code:trace}
> Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
>
[jira] [Commented] (COMPRESS-592) Checksum verification failed reading 7z archive with more than 65536 entries
[
https://issues.apache.org/jira/browse/COMPRESS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436069#comment-17436069
]
Roland Kreuzer commented on COMPRESS-592:
-
Just did a test run on one of the previously failing archives and can confirm
the issue fixed.
Thanks a lot, [~peterlee]
> Checksum verification failed reading 7z archive with more than 65536 entries
>
>
> Key: COMPRESS-592
> URL: https://issues.apache.org/jira/browse/COMPRESS-592
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.21
> Environment: Compress 1.21 and XZ 1.9 on JDK 11; reproduced on both
> Windows and Ubuntu Linux
>Reporter: Roland Kreuzer
>Priority: Major
> Attachments: _DOC.7z
>
>
> I have a use-case where I have to decompress Sevenzip archives from an
> external source which may have a large number of entries.
> I found decompression fails when trying to extract entry 65536 (zero-based
> index) with a checksum failure.
>
> I was able to reproduce the issue with a simple 7Zip file containing 70.001
> entries with random MD5 checksum textfiles (attached).
> The sample Archive was created using the 7Zip Windows client and uses
> LZMA2:3m.
>
> My code is a simple sequential read of all contents of the file like
> {code:java}
> @Test
> void readBigSevenZipFile() throws IOException
> {
> try (SevenZFile sevenZFile = new SevenZFile(new
> File("E:\\Temp\\_DOC.7z")))
> {
> SevenZArchiveEntry entry = sevenZFile.getNextEntry();
> while (entry != null)
> {
> if (entry.hasStream())
> {
> byte[] content = new byte[(int) entry.getSize()];
> sevenZFile.read(content);
> System.out.println(entry.getName());
> }
> entry = sevenZFile.getNextEntry();
> }
> }
> }
> {code}
> which fails consistently after file65535.txt with
> {code:java}
> java.io.IOException: Checksum verification failed
> at
> org.apache.commons.compress.utils.ChecksumVerifyingInputStream.read(ChecksumVerifyingInputStream.java:94)
> ~[commons-compress-1.21.jar!/:1.21]
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.read(SevenZFile.java:1905)
> ~[commons-compress-1.21.jar!/:1.21]
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.read(SevenZFile.java:1888)
> ~[commons-compress-1.21.jar!/:1.21]
> {code}
>
> It is noticeable that the value is 2 to the 16th power, which could suggest
> an overflow error of some sorts.
>
> While the minimal sample contains only small txt files, I originally found
> the issue with larger archives containing also Image and PDF files. The
> archive's contents or size in byte does not seem to have direct influence on
> the issue, only the number of files contained within.
>
> I did not find any workaround yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436000#comment-17436000
]
Gary D. Gregory edited comment on IO-755 at 10/29/21, 2:02 PM:
---
How about using {{PathUtils.walk(Path, PathFilter, int, boolean,
FileVisitOption...)}} and passing {{false}} to {{readAttributes}} which is what
{{FileUtils.streamFiles(File, boolean, String...)}} and consequently
{{FileUtils.iterateFiles(File, String[], boolean)}} and
{{FileUtils.listFiles(File, String[], boolean)}} do.
It would be nice if {{java.nio.file.FileVisitOption}} had a
{{DONT_READ_FILE_ATTRIBUTES}} value I suppose.
was (Author: garydgregory):
How about using {{PathUtils.walk(Path, PathFilter, int, boolean,
FileVisitOption...)}} and passing {{false}} to {{readAttributes}} which is what
{{FileUtils.streamFiles(File, boolean, String...)}} and consequently
{{FileUtils.iterateFiles(File, String[], boolean)}} and
{{FileUtils.listFiles(File, String[], boolean)}} do.
> Using FileUtils.listFiles with background changes fails on Linux
>
>
> Key: IO-755
> URL: https://issues.apache.org/jira/browse/IO-755
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Alexis Suter
>Priority: Major
> Labels: listFiles
>
> When calling {{FileUtils.listFiles}} and a file is being removed from this or
> a subdirectory during the execution the following Error occurs on linux.
> h3. Stacktrace
> {code:trace}
> Caused by: java.nio.file.NoSuchFileException: /**
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
> at
> java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
> at
> java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
> at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
> at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
> at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
> at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
> at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
> {code}
> h3. Upgrade of apache commons-io
> The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the
> library started using java.nio instead of java.io which might behave
> differently on unix, when a file is not existent during processing of
> directories.
> h3. Causing functionality in our implementation
> In some part we have multithreaded access to the filesystem. A folder is
> accessed from multiple threads which access the folder with
> {{org.apache.commons.io.FileUtils#listFiles(java.io.File,
> org.apache.commons.io.filefilter.IOFileFilter,
> org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
> *delete* and *rename* files from this directory. While this seems to be
> working on a windows machine, it is not the case on a Linux machine. In 2.8
> apparently there was no issue with this.
> *In short*: While a directory is being listed with {{FileUtils.listFiles()}},
> a file is being deleted. When trying to read its file-attributes it stops
> because of a {{NoSuchFileException}}.
> h3. Other commons-io functionalities
> We tried using {{PathUtils.walk(directory filters, 5,
> false).map(Path::toFile).collect(Collectors.toList())}}. This method was
> introduced with the same change.
> This produced a similar stacktrace:
> {code:trace}
> Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
>
[jira] [Commented] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436000#comment-17436000
]
Gary D. Gregory commented on IO-755:
How about using {{PathUtils.walk(Path, PathFilter, int, boolean,
FileVisitOption...)}} and passing {{false}} to {{readAttributes}} which is what
{{FileUtils.streamFiles(File, boolean, String...)}} and consequently
{{FileUtils.iterateFiles(File, String[], boolean)}} and
{{FileUtils.listFiles(File, String[], boolean)}} do.
> Using FileUtils.listFiles with background changes fails on Linux
>
>
> Key: IO-755
> URL: https://issues.apache.org/jira/browse/IO-755
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Alexis Suter
>Priority: Major
> Labels: listFiles
>
> When calling {{FileUtils.listFiles}} and a file is being removed from this or
> a subdirectory during the execution the following Error occurs on linux.
> h3. Stacktrace
> {code:trace}
> Caused by: java.nio.file.NoSuchFileException: /**
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
> at
> java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
> at
> java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
> at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
> at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
> at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
> at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
> at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
> {code}
> h3. Upgrade of apache commons-io
> The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the
> library started using java.nio instead of java.io which might behave
> differently on unix, when a file is not existent during processing of
> directories.
> h3. Causing functionality in our implementation
> In some part we have multithreaded access to the filesystem. A folder is
> accessed from multiple threads which access the folder with
> {{org.apache.commons.io.FileUtils#listFiles(java.io.File,
> org.apache.commons.io.filefilter.IOFileFilter,
> org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
> *delete* and *rename* files from this directory. While this seems to be
> working on a windows machine, it is not the case on a Linux machine. In 2.8
> apparently there was no issue with this.
> *In short*: While a directory is being listed with {{FileUtils.listFiles()}},
> a file is being deleted. When trying to read its file-attributes it stops
> because of a {{NoSuchFileException}}.
> h3. Other commons-io functionalities
> We tried using {{PathUtils.walk(directory filters, 5,
> false).map(Path::toFile).collect(Collectors.toList())}}. This method was
> introduced with the same change.
> This produced a similar stacktrace:
> {code:trace}
> Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
> ... 47
[jira] [Commented] (IO-754) WildcardFileFilter should not ignore override accept-method
[
https://issues.apache.org/jira/browse/IO-754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435976#comment-17435976
]
Gary D. Gregory commented on IO-754:
The intent was not to break existing apps of course. This is an unintended side
effect and I am sorry it has caused you this issue. If you can find a way to
still support old and new apps, then we would love to get a PR on GitHub, if
not we will take your advice and at least deprecate the method.
> WildcardFileFilter should not ignore override accept-method
> ---
>
> Key: IO-754
> URL: https://issues.apache.org/jira/browse/IO-754
> Project: Commons IO
> Issue Type: Bug
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Sita Geßner
>Priority: Major
>
> I have a method to filter files with different extensions in a directory.
> I did override the accept method for WildcardFileFilter because I want to
> check the filename in lowercase, to find files with different spellings of
> the fileextension.
> After updating commons-io from 2.8.0 to 2.11.0 the accept-method is ignored.
> The breakpoint won't stop in the accept-method.
> Here is my codeexample:
> {code:java}
> public static void main(final String[] args) throws Exception {
> final File directory = new File(FileUtils.getTempDirectory(), "TEST");
> directory.mkdir();
> new File(directory, "TEST").mkdir();
> new File(directory, "test1.pdf").createNewFile();
> new File(directory, "test1.txt").createNewFile();
> new File(directory, "test2.PDF").createNewFile();
> new File(directory, "test2.TXT").createNewFile();
> final IOFileFilter filter = new WildcardFileFilter("*.pdf", "*.txt") {
> private static final long serialVersionUID = 1L;
> @Override
> public boolean accept(final File file) {
> return super.accept(file, file.getName().toLowerCase());
> }
> };
> for (final Iterator itFiles = FileUtils.iterateFiles(directory,
> filter, null); itFiles
> .hasNext();) {
> final File file = itFiles.next();
> System.out.println(file.getAbsolutePath());
> }
> }
> {code}
> output in version 2.8.0:
> {noformat}
> /tmp/TEST/test2.PDF
> /tmp/TEST/test1.txt
> /tmp/TEST/test1.pdf
> /tmp/TEST/test2.TXT
> {noformat}
> output in version 2.9.0 or higher:
> {noformat}
> /tmp/TEST/test1.txt
> /tmp/TEST/test1.pdf
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IO-754) WildcardFileFilter should not ignore override accept-method
[
https://issues.apache.org/jira/browse/IO-754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435973#comment-17435973
]
Sita Geßner commented on IO-754:
Ok then the method accept(File)-method should be supported or throw an
UnsupportedOperationException in my opinion.
Also it should be marked as deprecated, so SonarLint would show this to me.
> WildcardFileFilter should not ignore override accept-method
> ---
>
> Key: IO-754
> URL: https://issues.apache.org/jira/browse/IO-754
> Project: Commons IO
> Issue Type: Bug
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Sita Geßner
>Priority: Major
>
> I have a method to filter files with different extensions in a directory.
> I did override the accept method for WildcardFileFilter because I want to
> check the filename in lowercase, to find files with different spellings of
> the fileextension.
> After updating commons-io from 2.8.0 to 2.11.0 the accept-method is ignored.
> The breakpoint won't stop in the accept-method.
> Here is my codeexample:
> {code:java}
> public static void main(final String[] args) throws Exception {
> final File directory = new File(FileUtils.getTempDirectory(), "TEST");
> directory.mkdir();
> new File(directory, "TEST").mkdir();
> new File(directory, "test1.pdf").createNewFile();
> new File(directory, "test1.txt").createNewFile();
> new File(directory, "test2.PDF").createNewFile();
> new File(directory, "test2.TXT").createNewFile();
> final IOFileFilter filter = new WildcardFileFilter("*.pdf", "*.txt") {
> private static final long serialVersionUID = 1L;
> @Override
> public boolean accept(final File file) {
> return super.accept(file, file.getName().toLowerCase());
> }
> };
> for (final Iterator itFiles = FileUtils.iterateFiles(directory,
> filter, null); itFiles
> .hasNext();) {
> final File file = itFiles.next();
> System.out.println(file.getAbsolutePath());
> }
> }
> {code}
> output in version 2.8.0:
> {noformat}
> /tmp/TEST/test2.PDF
> /tmp/TEST/test1.txt
> /tmp/TEST/test1.pdf
> /tmp/TEST/test2.TXT
> {noformat}
> output in version 2.9.0 or higher:
> {noformat}
> /tmp/TEST/test1.txt
> /tmp/TEST/test1.pdf
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435968#comment-17435968
]
Gary D. Gregory commented on IO-755:
WRT exceptions, unchecked exceptions make using lambdas easy.
> Using FileUtils.listFiles with background changes fails on Linux
>
>
> Key: IO-755
> URL: https://issues.apache.org/jira/browse/IO-755
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Alexis Suter
>Priority: Major
> Labels: listFiles
>
> When calling {{FileUtils.listFiles}} and a file is being removed from this or
> a subdirectory during the execution the following Error occurs on linux.
> h3. Stacktrace
> {code:trace}
> Caused by: java.nio.file.NoSuchFileException: /**
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
> at
> java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
> at
> java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
> at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
> at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
> at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
> at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
> at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
> {code}
> h3. Upgrade of apache commons-io
> The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the
> library started using java.nio instead of java.io which might behave
> differently on unix, when a file is not existent during processing of
> directories.
> h3. Causing functionality in our implementation
> In some part we have multithreaded access to the filesystem. A folder is
> accessed from multiple threads which access the folder with
> {{org.apache.commons.io.FileUtils#listFiles(java.io.File,
> org.apache.commons.io.filefilter.IOFileFilter,
> org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
> *delete* and *rename* files from this directory. While this seems to be
> working on a windows machine, it is not the case on a Linux machine. In 2.8
> apparently there was no issue with this.
> *In short*: While a directory is being listed with {{FileUtils.listFiles()}},
> a file is being deleted. When trying to read its file-attributes it stops
> because of a {{NoSuchFileException}}.
> h3. Other commons-io functionalities
> We tried using {{PathUtils.walk(directory filters, 5,
> false).map(Path::toFile).collect(Collectors.toList())}}. This method was
> introduced with the same change.
> This produced a similar stacktrace:
> {code:trace}
> Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
> ... 47 more
> Caused by: java.nio.file.NoSuchFileException: /*
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
>
[jira] [Commented] (IO-754) WildcardFileFilter should not ignore override accept-method
[
https://issues.apache.org/jira/browse/IO-754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435967#comment-17435967
]
Gary D. Gregory commented on IO-754:
We will not remove a method from an interface within a major release as this
would break binary compatibility.
> WildcardFileFilter should not ignore override accept-method
> ---
>
> Key: IO-754
> URL: https://issues.apache.org/jira/browse/IO-754
> Project: Commons IO
> Issue Type: Bug
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Sita Geßner
>Priority: Major
>
> I have a method to filter files with different extensions in a directory.
> I did override the accept method for WildcardFileFilter because I want to
> check the filename in lowercase, to find files with different spellings of
> the fileextension.
> After updating commons-io from 2.8.0 to 2.11.0 the accept-method is ignored.
> The breakpoint won't stop in the accept-method.
> Here is my codeexample:
> {code:java}
> public static void main(final String[] args) throws Exception {
> final File directory = new File(FileUtils.getTempDirectory(), "TEST");
> directory.mkdir();
> new File(directory, "TEST").mkdir();
> new File(directory, "test1.pdf").createNewFile();
> new File(directory, "test1.txt").createNewFile();
> new File(directory, "test2.PDF").createNewFile();
> new File(directory, "test2.TXT").createNewFile();
> final IOFileFilter filter = new WildcardFileFilter("*.pdf", "*.txt") {
> private static final long serialVersionUID = 1L;
> @Override
> public boolean accept(final File file) {
> return super.accept(file, file.getName().toLowerCase());
> }
> };
> for (final Iterator itFiles = FileUtils.iterateFiles(directory,
> filter, null); itFiles
> .hasNext();) {
> final File file = itFiles.next();
> System.out.println(file.getAbsolutePath());
> }
> }
> {code}
> output in version 2.8.0:
> {noformat}
> /tmp/TEST/test2.PDF
> /tmp/TEST/test1.txt
> /tmp/TEST/test1.pdf
> /tmp/TEST/test2.TXT
> {noformat}
> output in version 2.9.0 or higher:
> {noformat}
> /tmp/TEST/test1.txt
> /tmp/TEST/test1.pdf
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-parent] dependabot[bot] commented on pull request #67: Bump spotbugs-maven-plugin from 4.3.0 to 4.4.2.1
dependabot[bot] commented on pull request #67: URL: https://github.com/apache/commons-parent/pull/67#issuecomment-954666830 Superseded by #69. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [commons-parent] dependabot[bot] closed pull request #67: Bump spotbugs-maven-plugin from 4.3.0 to 4.4.2.1
dependabot[bot] closed pull request #67: URL: https://github.com/apache/commons-parent/pull/67 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [commons-parent] dependabot[bot] opened a new pull request #69: Bump spotbugs-maven-plugin from 4.3.0 to 4.4.2.2
dependabot[bot] opened a new pull request #69: URL: https://github.com/apache/commons-parent/pull/69 Bumps [spotbugs-maven-plugin](https://github.com/spotbugs/spotbugs-maven-plugin) from 4.3.0 to 4.4.2.2. Release notes Sourced from https://github.com/spotbugs/spotbugs-maven-plugin/releases;>spotbugs-maven-plugin's releases. Spotbugs-maven-plugin 4.4.2.2 Use new base-parent pom with removal of undocumented maven url attributes that cause issues for users of older jfrog artifactory installations. Spotbugs-maven-plugin 4.4.2.1 Release Reworked version string to account for any patches we need to make to plugin that would otherwise case a diverge from spotbugs or require us to wait. This is similar to how other plugins approach this such as lombok. The first 3 positions are reserved for the alignment with spotbugs. The last position is for our patch revision level. Normally this would be '0' but given we released 4.4.2 already, it made sense to denote '1' so that it was clear there was a difference. This patch release addresses issues with resolution of the maven dependencies that resulted in a few regression libraries that had vulnerabilities. This patch further changed lowest maven from 3.2.5 to 3.3.9 but reality is that even 3.3.9 likely doesn't work. Since all maven before 3.8.1 are vulnerable, most should be there. If not, let us know. Future releases will raise that revision number up. Spotbugs-maven-plugin 4.4.2 Release Added support for spotbugs 4.4.2 Now running github actions on jdk 17 and 18-ea to show this works there Now running against maven 3.8.3 Updated a number of plugins and dependencies Spotbugs-maven-plugin 4.4.1 Release Add support for Sarif Support spotbugs 4.4.1 Library Updates Commits https://github.com/spotbugs/spotbugs-maven-plugin/commit/a00e29d56997b9f4247748355a549e837c304629;>a00e29d [maven-release-plugin] prepare release spotbugs-maven-plugin-4.4.2.2 https://github.com/spotbugs/spotbugs-maven-plugin/commit/8c16d9704f00193c736d385b599496b64bb86880;>8c16d97 [pom] Update base parent without maven url attributes that break older artifa... https://github.com/spotbugs/spotbugs-maven-plugin/commit/7e4d513d8b6605dbdcd6edf7da8c05a34017654d;>7e4d513 Merge pull request https://github-redirect.dependabot.com/spotbugs/spotbugs-maven-plugin/issues/366;>#366 from spotbugs/dependabot/maven/org.apache.ant-ant-1.1... https://github.com/spotbugs/spotbugs-maven-plugin/commit/cdcecc47ac78d4efa1f0beedc139e11b45ca4168;>cdcecc4 Bump ant from 1.10.11 to 1.10.12 https://github.com/spotbugs/spotbugs-maven-plugin/commit/2b57e82ca7c5eadcde1f4479436ef53a1317f685;>2b57e82 [maven-release-plugin] prepare for next development iteration https://github.com/spotbugs/spotbugs-maven-plugin/commit/20bbdd5e9630f423d9943a8ef46879c4b9a0efac;>20bbdd5 [maven-release-plugin] prepare release spotbugs-maven-plugin-4.4.2.1 https://github.com/spotbugs/spotbugs-maven-plugin/commit/ea259b0fe3334a6ddc297218c710ef258728e462;>ea259b0 Merge pull request https://github-redirect.dependabot.com/spotbugs/spotbugs-maven-plugin/issues/365;>#365 from hazendaz/spotbugs https://github.com/spotbugs/spotbugs-maven-plugin/commit/9e2368119da25f2edcbd820bb25f18c716ddd5b5;>9e23681 [pom] Formatting https://github.com/spotbugs/spotbugs-maven-plugin/commit/bc67d6b48ef142b108395728ccda0e1b9588f057;>bc67d6b [pom] Override maven shared utils to 3.3.4 https://github.com/spotbugs/spotbugs-maven-plugin/commit/06a2e8b83a5649c3c3d087d66f541895e0fd5414;>06a2e8b [pom] Override commons lang2 to 2.6 Additional commits viewable in https://github.com/spotbugs/spotbugs-maven-plugin/compare/spotbugs-maven-plugin-4.3.0...spotbugs-maven-plugin-4.4.2.2;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot
[jira] [Updated] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Suter updated IO-755:
Affects Version/s: 2.9.0
2.10.0
2.11.0
> Using FileUtils.listFiles with background changes fails on Linux
>
>
> Key: IO-755
> URL: https://issues.apache.org/jira/browse/IO-755
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Alexis Suter
>Priority: Major
> Labels: listFiles
>
> When calling {{FileUtils.listFiles}} and a file is being removed from this or
> a subdirectory during the execution the following Error occurs on linux.
> h3. Stacktrace
> {code:trace}
> Caused by: java.nio.file.NoSuchFileException: /**
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
> at
> java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
> at
> java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
> at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
> at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
> at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
> at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
> at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
> {code}
> h3. Upgrade of apache commons-io
> The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the
> library started using java.nio instead of java.io which might behave
> differently on unix, when a file is not existent during processing of
> directories.
> h3. Causing functionality in our implementation
> In some part we have multithreaded access to the filesystem. A folder is
> accessed from multiple threads which access the folder with
> {{org.apache.commons.io.FileUtils#listFiles(java.io.File,
> org.apache.commons.io.filefilter.IOFileFilter,
> org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
> *delete* and *rename* files from this directory. While this seems to be
> working on a windows machine, it is not the case on a Linux machine. In 2.8
> apparently there was no issue with this.
> *In short*: While a directory is being listed with {{FileUtils.listFiles()}},
> a file is being deleted. When trying to read its file-attributes it stops
> because of a {{NoSuchFileException}}.
> h3. Other commons-io functionalities
> We tried using {{PathUtils.walk(directory filters, 5,
> false).map(Path::toFile).collect(Collectors.toList())}}. This method was
> introduced with the same change.
> This produced a similar stacktrace:
> {code:trace}
> Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
> ... 47 more
> Caused by: java.nio.file.NoSuchFileException: /*
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
>
[jira] [Updated] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Suter updated IO-755:
Component/s: Utilities
> Using FileUtils.listFiles with background changes fails on Linux
>
>
> Key: IO-755
> URL: https://issues.apache.org/jira/browse/IO-755
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
>Reporter: Alexis Suter
>Priority: Major
> Labels: listFiles
>
> When calling {{FileUtils.listFiles}} and a file is being removed from this or
> a subdirectory during the execution the following Error occurs on linux.
> h3. Stacktrace
> {code:trace}
> Caused by: java.nio.file.NoSuchFileException: /**
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
> at
> java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
> at
> java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
> at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
> at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
> at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
> at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
> at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
> {code}
> h3. Upgrade of apache commons-io
> The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the
> library started using java.nio instead of java.io which might behave
> differently on unix, when a file is not existent during processing of
> directories.
> h3. Causing functionality in our implementation
> In some part we have multithreaded access to the filesystem. A folder is
> accessed from multiple threads which access the folder with
> {{org.apache.commons.io.FileUtils#listFiles(java.io.File,
> org.apache.commons.io.filefilter.IOFileFilter,
> org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
> *delete* and *rename* files from this directory. While this seems to be
> working on a windows machine, it is not the case on a Linux machine. In 2.8
> apparently there was no issue with this.
> *In short*: While a directory is being listed with {{FileUtils.listFiles()}},
> a file is being deleted. When trying to read its file-attributes it stops
> because of a {{NoSuchFileException}}.
> h3. Other commons-io functionalities
> We tried using {{PathUtils.walk(directory filters, 5,
> false).map(Path::toFile).collect(Collectors.toList())}}. This method was
> introduced with the same change.
> This produced a similar stacktrace:
> {code:trace}
> Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
> ... 47 more
> Caused by: java.nio.file.NoSuchFileException: /*
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
>
[jira] [Reopened] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Suter reopened IO-755:
-
Had to discuss something with a colleague to be sure of the issue.
> Using FileUtils.listFiles with background changes fails on Linux
>
>
> Key: IO-755
> URL: https://issues.apache.org/jira/browse/IO-755
> Project: Commons IO
> Issue Type: Bug
>Reporter: Alexis Suter
>Priority: Major
>
> When calling {{FileUtils.listFiles}} and a file is being removed from this or
> a subdirectory during the execution the following Error occurs on linux.
> h3. Stacktrace
> {code:trace}
> Caused by: java.nio.file.NoSuchFileException: /**
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
> at
> java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
> at
> java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
> at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
> at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
> at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
> at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
> at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
> {code}
> h3. Upgrade of apache commons-io
> The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the
> library started using java.nio instead of java.io which might behave
> differently on unix, when a file is not existent during processing of
> directories.
> h3. Causing functionality in our implementation
> In some part we have multithreaded access to the filesystem. A folder is
> accessed from multiple threads which access the folder with
> {{org.apache.commons.io.FileUtils#listFiles(java.io.File,
> org.apache.commons.io.filefilter.IOFileFilter,
> org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
> *delete* and *rename* files from this directory. While this seems to be
> working on a windows machine, it is not the case on a Linux machine. In 2.8
> apparently there was no issue with this.
> *In short*: While a directory is being listed with {{FileUtils.listFiles()}},
> a file is being deleted. When trying to read its file-attributes it stops
> because of a {{NoSuchFileException}}.
> h3. Other commons-io functionalities
> We tried using {{PathUtils.walk(directory filters, 5,
> false).map(Path::toFile).collect(Collectors.toList())}}. This method was
> introduced with the same change.
> This produced a similar stacktrace:
> {code:trace}
> Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
> ... 47 more
> Caused by: java.nio.file.NoSuchFileException: /*
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
>
[jira] [Updated] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Suter updated IO-755:
Labels: listFiles (was: )
> Using FileUtils.listFiles with background changes fails on Linux
>
>
> Key: IO-755
> URL: https://issues.apache.org/jira/browse/IO-755
> Project: Commons IO
> Issue Type: Bug
>Reporter: Alexis Suter
>Priority: Major
> Labels: listFiles
>
> When calling {{FileUtils.listFiles}} and a file is being removed from this or
> a subdirectory during the execution the following Error occurs on linux.
> h3. Stacktrace
> {code:trace}
> Caused by: java.nio.file.NoSuchFileException: /**
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
> at
> java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
> at
> java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
> at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
> at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
> at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
> at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
> at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
> {code}
> h3. Upgrade of apache commons-io
> The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the
> library started using java.nio instead of java.io which might behave
> differently on unix, when a file is not existent during processing of
> directories.
> h3. Causing functionality in our implementation
> In some part we have multithreaded access to the filesystem. A folder is
> accessed from multiple threads which access the folder with
> {{org.apache.commons.io.FileUtils#listFiles(java.io.File,
> org.apache.commons.io.filefilter.IOFileFilter,
> org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
> *delete* and *rename* files from this directory. While this seems to be
> working on a windows machine, it is not the case on a Linux machine. In 2.8
> apparently there was no issue with this.
> *In short*: While a directory is being listed with {{FileUtils.listFiles()}},
> a file is being deleted. When trying to read its file-attributes it stops
> because of a {{NoSuchFileException}}.
> h3. Other commons-io functionalities
> We tried using {{PathUtils.walk(directory filters, 5,
> false).map(Path::toFile).collect(Collectors.toList())}}. This method was
> introduced with the same change.
> This produced a similar stacktrace:
> {code:trace}
> Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
> at
> java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
> at
> java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
> at
> ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
> ... 47 more
> Caused by: java.nio.file.NoSuchFileException: /*
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
>
[jira] [Updated] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Suter updated IO-755:
Description:
When calling {{FileUtils.listFiles}} and a file is being removed from this or a
subdirectory during the execution the following Error occurs on linux.
h3. Stacktrace
{code:trace}
Caused by: java.nio.file.NoSuchFileException: /**
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
at
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
{code}
h3. Upgrade of apache commons-io
The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the library
started using java.nio instead of java.io which might behave differently on
unix, when a file is not existent during processing of directories.
h3. Causing functionality in our implementation
In some part we have multithreaded access to the filesystem. A folder is
accessed from multiple threads which access the folder with
{{org.apache.commons.io.FileUtils#listFiles(java.io.File,
org.apache.commons.io.filefilter.IOFileFilter,
org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
*delete* and *rename* files from this directory. While this seems to be working
on a windows machine, it is not the case on a Linux machine. In 2.8 apparently
there was no issue with this.
*In short*: While a directory is being listed with {{FileUtils.listFiles()}}, a
file is being deleted. When trying to read its file-attributes it stops because
of a {{NoSuchFileException}}.
h3. Other commons-io functionalities
We tried using {{PathUtils.walk(directory filters, 5,
false).map(Path::toFile).collect(Collectors.toList())}}. This method was
introduced with the same change.
This produced a similar stacktrace:
{code:trace}
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
at
java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
at
java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
at
java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
at
ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
at
ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
... 47 more
Caused by: java.nio.file.NoSuchFileException: /*
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
at
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at
[jira] [Updated] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Suter updated IO-755:
Description:
When calling {{FileUtils.listFiles}} and a file is being removed from this or a
subdirectory during the execution the following Error occurs on linux.
h3. Stacktrace
{code:trace}
Caused by: java.nio.file.NoSuchFileException: /**
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
at
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
{code}
h3. Upgrade of apache commons-io
The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the library
started using java.nio instead of java.io which might behave differently on
unix, when a file is not existent during processing of directories.
h3. Causing functionality in our implementation
In some part we have multithreaded access to the filesystem. A folder is
accessed from multiple threads which access the folder with
{{org.apache.commons.io.FileUtils#listFiles(java.io.File,
org.apache.commons.io.filefilter.IOFileFilter,
org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
*delete* and *rename* files from this directory. While this seems to be working
on a windows machine, it is not the case on a Linux machine. In 2.8 apparently
there was no issue with this.
*In short*: While a directory is being listed with {{FileUtils.listFiles()}}, a
file is being deleted. When trying to read its file-attributes it stops because
of a {{NoSuchFileException}}.
h3. Other commons-io functionalities
We tried using {{PathUtils.walk(directory filters, 5,
false).map(Path::toFile).collect(Collectors.toList())}}. This method was
introduced with the same change.
This produced a similar stacktrace:
{code:trace}
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
at
java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
at
java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
at
java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
at
ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
at
ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
... 47 more
Caused by: java.nio.file.NoSuchFileException: /*
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
at
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at
[jira] [Updated] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Suter updated IO-755:
Description:
When calling {{FileUtils.listFiles}} and a file is being removed from this or a
subdirectory during the execution the following Error occurs on linux.
h3. Stacktrace
{code:trace}
Caused by: java.nio.file.NoSuchFileException: /**
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
at
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
{code}
h3. Upgrade of apache commons-io
The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the library
started using java.nio instead of java.io which might behave differently on
unix, when a file is not existent during processing of directories.
h3. Causing functionality in our implementation
In some part we have multithreaded access to the filesystem. A folder is
accessed from multiple threads which access the folder with
{{org.apache.commons.io.FileUtils#listFiles(java.io.File,
org.apache.commons.io.filefilter.IOFileFilter,
org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
*delete* and *rename* files from this directory. While this seems to be working
on a windows machine, it is not the case on a Linux machine. In 2.8 apparently
there was no issue with this.
*In short*: While a directory is being listed with {{FileUtils.listFiles()}}, a
file is being deleted. When trying to read its file-attributes it stops because
of a {{NoSuchFileException}}.
h3. Other commons-io functionalities
We tried using {{PathUtils.walk(directory filters, 5,
false).map(Path::toFile).collect(Collectors.toList())}}
which produced a similar stacktrace:
{code:trace}
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
at
java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
at
java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
at
java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
at
ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
at
ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
... 47 more
Caused by: java.nio.file.NoSuchFileException: /*
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
at
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
[jira] [Updated] (IO-755) Using FileUtils.listFiles with background changes fails on Linux
[
https://issues.apache.org/jira/browse/IO-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Suter updated IO-755:
Description:
h3. Stacktrace
{code:trace}
Caused by: java.nio.file.NoSuchFileException: /**
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
at
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
{code}
h3. Upgrade of apache commons-io
The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the library
started using java.nio instead of java.io which might behave differently on
unix, when a file is not existent during processing of directories.
h3. Causing functionality in our implementation
In some part we have multithreaded access to the filesystem. A folder is
accessed from multiple threads which access the folder with
{{org.apache.commons.io.FileUtils#listFiles(java.io.File,
org.apache.commons.io.filefilter.IOFileFilter,
org.apache.commons.io.filefilter.IOFileFilter)}}. These threads also *move*,
*delete* and *rename* files from this directory. While this seems to be working
on a windows machine, it is not the case on a Linux machine. In 2.8 apparently
there was no issue with this.
*In short*: While a directory is beeing listed with {{FileUtils.listFiles()}},
a file is beeing deleted. When trying to read its file-attributes it stops
because of a {{NoSuchFileException}}.
h3. Other commons-io functionalities
We tried using {{PathUtils.walk(directory filters, 5,
false).map(Path::toFile).collect(Collectors.toList())}}
which produced a similar stacktrace:
{code:trace}
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
at
java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
at
java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
at
java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
ch.elca.ole.filenet.FilenetAccessorImpl.listFilesDocumentsStorageDir(FilenetAccessorImpl.java:439)
at
ch.elca.ole.filenet.FilenetAccessorImpl.buildStub(FilenetAccessorImpl.java:401)
at
ch.elca.ole.filenet.FilenetAccessorImpl.createStub(FilenetAccessorImpl.java:310)
... 47 more
Caused by: java.nio.file.NoSuchFileException: /*
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
at
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
at
java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:83)
... 59 more{code}
h3. Other Questions
* What
[jira] [Commented] (COMPRESS-592) Checksum verification failed reading 7z archive with more than 65536 entries
[
https://issues.apache.org/jira/browse/COMPRESS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435890#comment-17435890
]
Peter Lee commented on COMPRESS-592:
I pushed a fix. Please have a look. [~rolandkreuzer]
> Checksum verification failed reading 7z archive with more than 65536 entries
>
>
> Key: COMPRESS-592
> URL: https://issues.apache.org/jira/browse/COMPRESS-592
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.21
> Environment: Compress 1.21 and XZ 1.9 on JDK 11; reproduced on both
> Windows and Ubuntu Linux
>Reporter: Roland Kreuzer
>Priority: Major
> Attachments: _DOC.7z
>
>
> I have a use-case where I have to decompress Sevenzip archives from an
> external source which may have a large number of entries.
> I found decompression fails when trying to extract entry 65536 (zero-based
> index) with a checksum failure.
>
> I was able to reproduce the issue with a simple 7Zip file containing 70.001
> entries with random MD5 checksum textfiles (attached).
> The sample Archive was created using the 7Zip Windows client and uses
> LZMA2:3m.
>
> My code is a simple sequential read of all contents of the file like
> {code:java}
> @Test
> void readBigSevenZipFile() throws IOException
> {
> try (SevenZFile sevenZFile = new SevenZFile(new
> File("E:\\Temp\\_DOC.7z")))
> {
> SevenZArchiveEntry entry = sevenZFile.getNextEntry();
> while (entry != null)
> {
> if (entry.hasStream())
> {
> byte[] content = new byte[(int) entry.getSize()];
> sevenZFile.read(content);
> System.out.println(entry.getName());
> }
> entry = sevenZFile.getNextEntry();
> }
> }
> }
> {code}
> which fails consistently after file65535.txt with
> {code:java}
> java.io.IOException: Checksum verification failed
> at
> org.apache.commons.compress.utils.ChecksumVerifyingInputStream.read(ChecksumVerifyingInputStream.java:94)
> ~[commons-compress-1.21.jar!/:1.21]
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.read(SevenZFile.java:1905)
> ~[commons-compress-1.21.jar!/:1.21]
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.read(SevenZFile.java:1888)
> ~[commons-compress-1.21.jar!/:1.21]
> {code}
>
> It is noticeable that the value is 2 to the 16th power, which could suggest
> an overflow error of some sorts.
>
> While the minimal sample contains only small txt files, I originally found
> the issue with larger archives containing also Image and PDF files. The
> archive's contents or size in byte does not seem to have direct influence on
> the issue, only the number of files contained within.
>
> I did not find any workaround yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (COMPRESS-592) Checksum verification failed reading 7z archive with more than 65536 entries
[
https://issues.apache.org/jira/browse/COMPRESS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435881#comment-17435881
]
Peter Lee commented on COMPRESS-592:
Hi [~rolandkreuzer]
Thank you for your reporting! I think I have located this problem. Will try to
fix this soon.
> Checksum verification failed reading 7z archive with more than 65536 entries
>
>
> Key: COMPRESS-592
> URL: https://issues.apache.org/jira/browse/COMPRESS-592
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
>Affects Versions: 1.21
> Environment: Compress 1.21 and XZ 1.9 on JDK 11; reproduced on both
> Windows and Ubuntu Linux
>Reporter: Roland Kreuzer
>Priority: Major
> Attachments: _DOC.7z
>
>
> I have a use-case where I have to decompress Sevenzip archives from an
> external source which may have a large number of entries.
> I found decompression fails when trying to extract entry 65536 (zero-based
> index) with a checksum failure.
>
> I was able to reproduce the issue with a simple 7Zip file containing 70.001
> entries with random MD5 checksum textfiles (attached).
> The sample Archive was created using the 7Zip Windows client and uses
> LZMA2:3m.
>
> My code is a simple sequential read of all contents of the file like
> {code:java}
> @Test
> void readBigSevenZipFile() throws IOException
> {
> try (SevenZFile sevenZFile = new SevenZFile(new
> File("E:\\Temp\\_DOC.7z")))
> {
> SevenZArchiveEntry entry = sevenZFile.getNextEntry();
> while (entry != null)
> {
> if (entry.hasStream())
> {
> byte[] content = new byte[(int) entry.getSize()];
> sevenZFile.read(content);
> System.out.println(entry.getName());
> }
> entry = sevenZFile.getNextEntry();
> }
> }
> }
> {code}
> which fails consistently after file65535.txt with
> {code:java}
> java.io.IOException: Checksum verification failed
> at
> org.apache.commons.compress.utils.ChecksumVerifyingInputStream.read(ChecksumVerifyingInputStream.java:94)
> ~[commons-compress-1.21.jar!/:1.21]
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.read(SevenZFile.java:1905)
> ~[commons-compress-1.21.jar!/:1.21]
> at
> org.apache.commons.compress.archivers.sevenz.SevenZFile.read(SevenZFile.java:1888)
> ~[commons-compress-1.21.jar!/:1.21]
> {code}
>
> It is noticeable that the value is 2 to the 16th power, which could suggest
> an overflow error of some sorts.
>
> While the minimal sample contains only small txt files, I originally found
> the issue with larger archives containing also Image and PDF files. The
> archive's contents or size in byte does not seem to have direct influence on
> the issue, only the number of files contained within.
>
> I did not find any workaround yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IO-754) WildcardFileFilter should not ignore override accept-method
[
https://issues.apache.org/jira/browse/IO-754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435796#comment-17435796
]
Sita Geßner commented on IO-754:
[~ggregory] With this it's working for me. The breakpoint stops and the result
is like in version 2.8.0:
{code:java}
@Override
public FileVisitResult accept(final Path file, final
BasicFileAttributes attributes) {
if (super.accept(file.toFile(),
file.getFileName().toString().toLowerCase())) {
return FileVisitResult.CONTINUE;
}
return FileVisitResult.TERMINATE;
}
{code}
I think it would help if the accept(File)-method is supported in the next
version. If you don't like to support it anymore, it should be removed from the
interface.
> WildcardFileFilter should not ignore override accept-method
> ---
>
> Key: IO-754
> URL: https://issues.apache.org/jira/browse/IO-754
> Project: Commons IO
> Issue Type: Bug
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Sita Geßner
>Priority: Major
>
> I have a method to filter files with different extensions in a directory.
> I did override the accept method for WildcardFileFilter because I want to
> check the filename in lowercase, to find files with different spellings of
> the fileextension.
> After updating commons-io from 2.8.0 to 2.11.0 the accept-method is ignored.
> The breakpoint won't stop in the accept-method.
> Here is my codeexample:
> {code:java}
> public static void main(final String[] args) throws Exception {
> final File directory = new File(FileUtils.getTempDirectory(), "TEST");
> directory.mkdir();
> new File(directory, "TEST").mkdir();
> new File(directory, "test1.pdf").createNewFile();
> new File(directory, "test1.txt").createNewFile();
> new File(directory, "test2.PDF").createNewFile();
> new File(directory, "test2.TXT").createNewFile();
> final IOFileFilter filter = new WildcardFileFilter("*.pdf", "*.txt") {
> private static final long serialVersionUID = 1L;
> @Override
> public boolean accept(final File file) {
> return super.accept(file, file.getName().toLowerCase());
> }
> };
> for (final Iterator itFiles = FileUtils.iterateFiles(directory,
> filter, null); itFiles
> .hasNext();) {
> final File file = itFiles.next();
> System.out.println(file.getAbsolutePath());
> }
> }
> {code}
> output in version 2.8.0:
> {noformat}
> /tmp/TEST/test2.PDF
> /tmp/TEST/test1.txt
> /tmp/TEST/test1.pdf
> /tmp/TEST/test2.TXT
> {noformat}
> output in version 2.9.0 or higher:
> {noformat}
> /tmp/TEST/test1.txt
> /tmp/TEST/test1.pdf
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
