[jira] [Work logged] (IO-649) IOUtils contentEquals method performance improvements
[ https://issues.apache.org/jira/browse/IO-649?focusedWorklogId=382723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382723 ] ASF GitHub Bot logged work on IO-649: - Author: ASF GitHub Bot Created on: 06/Feb/20 04:57 Start Date: 06/Feb/20 04:57 Worklog Time Spent: 10m Work Description: brettlounsbury commented on issue #101: IO-649 - Improve the performance of the contentEquals() methods. URL: https://github.com/apache/commons-io/pull/101#issuecomment-582737191 @garydgregory - Any other feedback here or anything else I can do to improve this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382723) Time Spent: 5h 10m (was: 5h) > IOUtils contentEquals method performance improvements > - > > Key: IO-649 > URL: https://issues.apache.org/jira/browse/IO-649 > Project: Commons IO > Issue Type: Improvement > Components: Utilities >Affects Versions: 1.0, 1.1 >Reporter: Brett Lounsbury >Priority: Major > Fix For: 2.6 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > > contentEquals() internally wraps any given InputStream/Reader in a Buffered > version (if it is not already buffered) which avoids a lot of IO penalties, > but then it proceeds to read each byte/character one at a time. This leads > to significantly more method calls and also a lot of byte -> int casting > since the read() method returns an int between 0 and 255 instead of returning > a byte. > > I have a change that modifies the contentEquals() methods to internally > buffer content into a byte/char array and to then do batch comparisons of > those arrays using Arrays.equals instead of using a BufferedInputStream or > BufferedReader and making use of the single byte/char read() methods. This > reduces the number of method invocations by a factor equal to the buffer size > and avoids casting every byte read to an int. > > The following table shows the performance increase over 1000 iterations of > comparing 2 1GB InputStream of binary data (stored in memory to avoid I/O). > This test was performed on an EC2 M4.4XL host using Java 1.8.0.232 and there > was a forced System.gc() between each iteration to avoid GC as a source of > latency: > Average: 7236 to 858ms (8.43x speedup) > P50: 7224 to 856ms (8.44x speedup) > P90: 7249 to 860ms (8.43x speedup) > P99: 7410 to 913ms (8.12x speedup) > P100: 8330 to 1278ms (6.52x speedup) > > The following table shows the performance increase over 1000 iterations of > comparing 2 1GB Reader of character data (stored in memory to avoid I/O). > This test was performed on an EC2 M4.4XL host using Java 1.8.0.232 and there > was a forced System.gc() between each iteration to avoid GC as a source of > latency: > Average: 11281 to 1737ms (6.50x speedup) > P50: 11262 to 1735ms (6.49x speedup) > P90: 11292 to 1741ms (6.49x speedup) > P99: 11707 to 1774ms (6.60x speedup) > P100: 12176 to 1884ms (6.46x speedup) > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [commons-io] brettlounsbury commented on issue #101: IO-649 - Improve the performance of the contentEquals() methods.
brettlounsbury commented on issue #101: IO-649 - Improve the performance of the contentEquals() methods. URL: https://github.com/apache/commons-io/pull/101#issuecomment-582737191 @garydgregory - Any other feedback here or anything else I can do to improve this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NET-677) UTF-8 supporting in IMAP
Trần Tiến Đức created NET-677: - Summary: UTF-8 supporting in IMAP Key: NET-677 URL: https://issues.apache.org/jira/browse/NET-677 Project: Commons Net Issue Type: Bug Reporter: Trần Tiến Đức Hello, I found that the org.apache.commons.net.imap.IMAP uses hardcoded `__DEFAULT_ENCODING` which is "ISO-8859-1". Which means every lines read & write are decoded/encoded by "ISO-8859-1". I wonder is there any way to make my IMAPClient use UTF-8 encoding? BTW, I had experimented two attempts: - Create a new MyIMAPClient inheriting IMAPClient and override `_connectAction_()` to re-assign `_reader` and `_writer` to use UTF-8 encoding - Copy & Pasting the IMAP and the IMAPClient to my local project, change `__DEFAULT_ENCODING` to UTF-8 None of above works Duc Tran -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [commons-daemon] RayWang10 opened a new pull request #15: apsupport.m4: remove supported_os define when check host_cpu, it's
RayWang10 opened a new pull request #15: apsupport.m4: remove supported_os define when check host_cpu, it's URL: https://github.com/apache/commons-daemon/pull/15 defined at check host_os, and supported_os cannot be defined as cpu. fix build error, when configure without '--build --host' options on s390, arm, aarch64, mipsel, mips. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (VFS-753) NumberFormatException in SftpFileSystem::getUId
[ https://issues.apache.org/jira/browse/VFS-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031182#comment-17031182 ] Gary D. Gregory commented on VFS-753: - YW. Please make a note here if this addresses your issue in your actual testing (beyond reviewing in the code.) > NumberFormatException in SftpFileSystem::getUId > --- > > Key: VFS-753 > URL: https://issues.apache.org/jira/browse/VFS-753 > Project: Commons VFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: John Webb >Priority: Major > Fix For: 2.6.1 > > > When calling FileSystemManager::resolveFile(...) connecting to a specific > sftp server a NumberFormat exception is thrown and a FileObject is not > returned. > When the SftpFileSystem constructor is called there is a call to > detectExecDisabled which in turn calls getUid. SftpFileSystem::getUId assumes > that executeCommand(...) will always return non zero (the channel exit > status) for execution failure. > In the case of the sftp server that we are connecting to, the channel exit > status remains zero however the returned buffer contains the string "This > server does not provide shell access, only SFTP. Goodbye.". This causes a > NumberFormatException in getUId. > This behaviour does not occur in version 2.4.1 > Here is the relevent snippet from the stack trace: > java.lang.NumberFormatException: For input string: "This server does not > provide shell access, only SFTP. Goodbye." > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.getUId(SftpFileSystem.java:281) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.detectExecDisabled(SftpFileSystem.java:344) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.(SftpFileSystem.java:94) > at > org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:93) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:93) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:72) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:56) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:717) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:654) > at > com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtil.ls(SFTPUtil.java:68) > at > com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtilTest.testConnectToGoogle(SFTPUtilTest.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (VFS-753) NumberFormatException in SftpFileSystem::getUId
[
https://issues.apache.org/jira/browse/VFS-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031112#comment-17031112
]
John Webb commented on VFS-753:
---
[~ggregory]
The new test in {{detectExecDisabled}} looks good to me.
Thanks for you help mate.
> NumberFormatException in SftpFileSystem::getUId
> ---
>
> Key: VFS-753
> URL: https://issues.apache.org/jira/browse/VFS-753
> Project: Commons VFS
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: John Webb
>Priority: Major
> Fix For: 2.6.1
>
>
> When calling FileSystemManager::resolveFile(...) connecting to a specific
> sftp server a NumberFormat exception is thrown and a FileObject is not
> returned.
> When the SftpFileSystem constructor is called there is a call to
> detectExecDisabled which in turn calls getUid. SftpFileSystem::getUId assumes
> that executeCommand(...) will always return non zero (the channel exit
> status) for execution failure.
> In the case of the sftp server that we are connecting to, the channel exit
> status remains zero however the returned buffer contains the string "This
> server does not provide shell access, only SFTP. Goodbye.". This causes a
> NumberFormatException in getUId.
> This behaviour does not occur in version 2.4.1
> Here is the relevent snippet from the stack trace:
> java.lang.NumberFormatException: For input string: "This server does not
> provide shell access, only SFTP. Goodbye."
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Integer.parseInt(Integer.java:615)
> at
> org.apache.commons.vfs2.provider.sftp.SftpFileSystem.getUId(SftpFileSystem.java:281)
> at
> org.apache.commons.vfs2.provider.sftp.SftpFileSystem.detectExecDisabled(SftpFileSystem.java:344)
> at
> org.apache.commons.vfs2.provider.sftp.SftpFileSystem.(SftpFileSystem.java:94)
> at
> org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:93)
> at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:93)
> at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:72)
> at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:56)
> at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:717)
> at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:654)
> at
> com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtil.ls(SFTPUtil.java:68)
> at
> com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtilTest.testConnectToGoogle(SFTPUtilTest.java:48)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (VFS-753) NumberFormatException in SftpFileSystem::getUId
[
https://issues.apache.org/jira/browse/VFS-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031103#comment-17031103
]
Gary D. Gregory commented on VFS-753:
-
[~JohnWebb]
I adjusted the test in {{detectExecDisabled}} to check for a return value from
{{getUId()}} that corresponds to the internal call having failed.
If you can think of something better, please provide a PR on GitHub. If you
think this fix is still too risky, I can back it out.
> NumberFormatException in SftpFileSystem::getUId
> ---
>
> Key: VFS-753
> URL: https://issues.apache.org/jira/browse/VFS-753
> Project: Commons VFS
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: John Webb
>Priority: Major
> Fix For: 2.6.1
>
>
> When calling FileSystemManager::resolveFile(...) connecting to a specific
> sftp server a NumberFormat exception is thrown and a FileObject is not
> returned.
> When the SftpFileSystem constructor is called there is a call to
> detectExecDisabled which in turn calls getUid. SftpFileSystem::getUId assumes
> that executeCommand(...) will always return non zero (the channel exit
> status) for execution failure.
> In the case of the sftp server that we are connecting to, the channel exit
> status remains zero however the returned buffer contains the string "This
> server does not provide shell access, only SFTP. Goodbye.". This causes a
> NumberFormatException in getUId.
> This behaviour does not occur in version 2.4.1
> Here is the relevent snippet from the stack trace:
> java.lang.NumberFormatException: For input string: "This server does not
> provide shell access, only SFTP. Goodbye."
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Integer.parseInt(Integer.java:615)
> at
> org.apache.commons.vfs2.provider.sftp.SftpFileSystem.getUId(SftpFileSystem.java:281)
> at
> org.apache.commons.vfs2.provider.sftp.SftpFileSystem.detectExecDisabled(SftpFileSystem.java:344)
> at
> org.apache.commons.vfs2.provider.sftp.SftpFileSystem.(SftpFileSystem.java:94)
> at
> org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:93)
> at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:93)
> at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:72)
> at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:56)
> at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:717)
> at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:654)
> at
> com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtil.ls(SFTPUtil.java:68)
> at
> com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtilTest.testConnectToGoogle(SFTPUtilTest.java:48)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (VFS-753) NumberFormatException in SftpFileSystem::getUId
[ https://issues.apache.org/jira/browse/VFS-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031078#comment-17031078 ] John Webb commented on VFS-753: --- [~ggregory] I suspect the fix has the side effect of telling the class that execution is enabled on the server when it isn't. detectExecDisabled relies on a throw to return true however the fix simply catches and logs. I obviously don't know the code as well as you but worry that a false negative from detectExecDisabled may lead to undesirable behaviour. > NumberFormatException in SftpFileSystem::getUId > --- > > Key: VFS-753 > URL: https://issues.apache.org/jira/browse/VFS-753 > Project: Commons VFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: John Webb >Priority: Major > Fix For: 2.6.1 > > > When calling FileSystemManager::resolveFile(...) connecting to a specific > sftp server a NumberFormat exception is thrown and a FileObject is not > returned. > When the SftpFileSystem constructor is called there is a call to > detectExecDisabled which in turn calls getUid. SftpFileSystem::getUId assumes > that executeCommand(...) will always return non zero (the channel exit > status) for execution failure. > In the case of the sftp server that we are connecting to, the channel exit > status remains zero however the returned buffer contains the string "This > server does not provide shell access, only SFTP. Goodbye.". This causes a > NumberFormatException in getUId. > This behaviour does not occur in version 2.4.1 > Here is the relevent snippet from the stack trace: > java.lang.NumberFormatException: For input string: "This server does not > provide shell access, only SFTP. Goodbye." > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.getUId(SftpFileSystem.java:281) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.detectExecDisabled(SftpFileSystem.java:344) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.(SftpFileSystem.java:94) > at > org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:93) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:93) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:72) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:56) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:717) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:654) > at > com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtil.ls(SFTPUtil.java:68) > at > com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtilTest.testConnectToGoogle(SFTPUtilTest.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (COMMONSSITE-131) Add EventRouter to apache commons
Manfred Huber created COMMONSSITE-131: - Summary: Add EventRouter to apache commons Key: COMMONSSITE-131 URL: https://issues.apache.org/jira/browse/COMMONSSITE-131 Project: Commons All Issue Type: New Feature Components: Commons Build Reporter: Manfred Huber I often needed some sort of a tiny "Event Bus" system, but could not find anything similar in apache commons. So I coded my own version of what I understand as a "tiny event bus" and I thought it may be practical to move this to apache-commons. Maybe someone can have a look at my project on [GitHub|https://github.com/downdrown/eventrouter] and eventually I could patch it to fit into apache-commons. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JEXL-302) JexlScript.getVariables returns strange values for array access
[
https://issues.apache.org/jira/browse/JEXL-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030755#comment-17030755
]
Dmitri Blinov commented on JEXL-302:
Maybe I'm kicking a dead horse here, but can you please explain the logic
behind {{x.y[['z', 't']]}}
{code:java}
@Test
public void testLiteral() throws Exception {
JexlScript e = JEXL.createScript("x.y[['z', 't']]");
Set> vars = e.getVariables();
Assert.assertEquals(1, vars.size());
Assert.assertTrue(eq(mkref(new String[][]{{"x", "y", "[ 'z', 't' ]"}}),
vars));
} {code}
Why do we expect the last "[ 'z', 't' ]" part? How is it differ from, for
instance, {{x.y['z' + 't']}} in a sense that both {{'z' + 't'}} and {{['z',
't']}} are just expressions?
> JexlScript.getVariables returns strange values for array access
> ---
>
> Key: JEXL-302
> URL: https://issues.apache.org/jira/browse/JEXL-302
> Project: Commons JEXL
> Issue Type: Bug
>Affects Versions: 3.1
>Reporter: Dmitri Blinov
>Assignee: Henri Biestro
>Priority: Minor
> Fix For: 3.2
>
>
> I can not understand the logic behind the current implementation of
> {{JexlScript.getVariables()}} method. From the documentation we know that the
> result should be the set of script variables. For the code
> {code:java}
> a[b][c]{code}
> it gives three variables {{a}}, {{b}}, {{c}}. So far so good. But for the code
> {code:java}
> a[b]['c']{code}
> it returns {{a}} and {{b c}}, where second variable has two fragments {{b}}
> and {{c}}. The documentation states that variables with multiple fragments
> are ant-ish variables, but I don't have any of ant-ish variables in the
> example, and {{'c'}} is not a variable, but a constant. I expect to get {{a}}
> and {{b}} as a result.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-vfs] garydgregory commented on a change in pull request #80: sftp add possibility to configure whether exec is enabled
garydgregory commented on a change in pull request #80: sftp add possibility to
configure whether exec is enabled
URL: https://github.com/apache/commons-vfs/pull/80#discussion_r375335332
##
File path:
commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/sftp/SftpFileSystem.java
##
@@ -91,7 +91,13 @@ protected SftpFileSystem(final GenericFileName rootName,
final Session session,
this.session = Objects.requireNonNull(session, "session");
this.connectTimeoutMillis = SftpFileSystemConfigBuilder.getInstance()
.getConnectTimeoutMillis(fileSystemOptions);
-this.execDisabled = detectExecDisabled();
+
+if
(SftpFileSystemConfigBuilder.getInstance().isExecEnabled(fileSystemOptions)){
+this.execDisabled = detectExecDisabled();
+}
Review comment:
formatting should be "} else {"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [commons-vfs] garydgregory commented on issue #80: sftp add possibility to configure whether exec is enabled
garydgregory commented on issue #80: sftp add possibility to configure whether exec is enabled URL: https://github.com/apache/commons-vfs/pull/80#issuecomment-582472307 @ddg-igh , Thank you for your PR changes. Please see my comments. Also, please rebase on master, I just fixed an issue in this provider. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [commons-vfs] garydgregory commented on a change in pull request #80: sftp add possibility to configure whether exec is enabled
garydgregory commented on a change in pull request #80: sftp add possibility to
configure whether exec is enabled
URL: https://github.com/apache/commons-vfs/pull/80#discussion_r375337947
##
File path:
commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/sftp/SftpFileSystem.java
##
@@ -91,7 +91,13 @@ protected SftpFileSystem(final GenericFileName rootName,
final Session session,
this.session = Objects.requireNonNull(session, "session");
this.connectTimeoutMillis = SftpFileSystemConfigBuilder.getInstance()
.getConnectTimeoutMillis(fileSystemOptions);
-this.execDisabled = detectExecDisabled();
+
+if
(SftpFileSystemConfigBuilder.getInstance().isExecEnabled(fileSystemOptions)){
+this.execDisabled = detectExecDisabled();
Review comment:
This feature disables _detection_ of the exec channel, so it should be
called as such IMO: `EXEC_ENABLED` -> `DISABLE_DETECT_EXEC_CHANEL`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [commons-vfs] garydgregory commented on a change in pull request #80: sftp add possibility to configure whether exec is enabled
garydgregory commented on a change in pull request #80: sftp add possibility to
configure whether exec is enabled
URL: https://github.com/apache/commons-vfs/pull/80#discussion_r375335941
##
File path:
commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/sftp/SftpFileSystemConfigBuilder.java
##
@@ -268,6 +269,21 @@ public ConfigRepository getConfigRepository(final
FileSystemOptions opts) {
public boolean isLoadOpenSSHConfig(final FileSystemOptions opts) {
return this.getBoolean(opts, LOAD_OPENSSH_CONFIG, Boolean.FALSE);
}
+
+/**
+ * Returns {@link Boolean#TRUE} if exec should be tested and enabled.
Review comment:
The API returns a `boolean`, not a `Boolean`, so the Javadoc should refer to
`true`, not `Boolean#TRUE}.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (VFS-753) NumberFormatException in SftpFileSystem::getUId
[ https://issues.apache.org/jira/browse/VFS-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary D. Gregory resolved VFS-753. - Fix Version/s: 2.6.1 Resolution: Fixed [~JohnWebb] Thank you for your report. Please try the latest from git or a snapshot build from https://repository.apache.org/content/repositories/snapshots/org/apache/commons/ > NumberFormatException in SftpFileSystem::getUId > --- > > Key: VFS-753 > URL: https://issues.apache.org/jira/browse/VFS-753 > Project: Commons VFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: John Webb >Priority: Major > Fix For: 2.6.1 > > > When calling FileSystemManager::resolveFile(...) connecting to a specific > sftp server a NumberFormat exception is thrown and a FileObject is not > returned. > When the SftpFileSystem constructor is called there is a call to > detectExecDisabled which in turn calls getUid. SftpFileSystem::getUId assumes > that executeCommand(...) will always return non zero (the channel exit > status) for execution failure. > In the case of the sftp server that we are connecting to, the channel exit > status remains zero however the returned buffer contains the string "This > server does not provide shell access, only SFTP. Goodbye.". This causes a > NumberFormatException in getUId. > This behaviour does not occur in version 2.4.1 > Here is the relevent snippet from the stack trace: > java.lang.NumberFormatException: For input string: "This server does not > provide shell access, only SFTP. Goodbye." > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.getUId(SftpFileSystem.java:281) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.detectExecDisabled(SftpFileSystem.java:344) > at > org.apache.commons.vfs2.provider.sftp.SftpFileSystem.(SftpFileSystem.java:94) > at > org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:93) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:93) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:72) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:56) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:717) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:654) > at > com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtil.ls(SFTPUtil.java:68) > at > com.shortcuts.lambda.google_reserve_sftp.util.SFTPUtilTest.testConnectToGoogle(SFTPUtilTest.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [commons-vfs] ddg-igh opened a new pull request #80: sftp add possibility to configure whether exec is enabled
ddg-igh opened a new pull request #80: sftp add possibility to configure whether exec is enabled URL: https://github.com/apache/commons-vfs/pull/80 One of our customer has an interesting sftp server. The server does not allow any further channels to be established after the execution of the exec command (getUid()). So it is not possible to do any filesystem operations, because at first the exec command is executed to test if exec is disabled. Maybe the server things we try to break the security or something else. So my idea to solve this problem is to make execDisabled configurable. This is done here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [commons-vfs] ddg-igh closed pull request #79: add possibility to configure if exec for sftp is disabled
ddg-igh closed pull request #79: add possibility to configure if exec for sftp is disabled URL: https://github.com/apache/commons-vfs/pull/79 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [commons-vfs] garydgregory commented on issue #79: add possibility to configure if exec for sftp is disabled
garydgregory commented on issue #79: add possibility to configure if exec for sftp is disabled URL: https://github.com/apache/commons-vfs/pull/79#issuecomment-582451950 @ddg-igh -1 You need to resubmit this PR as _every single line_ in SftpFileSystem.java is marked as changed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [commons-vfs] ddg-igh opened a new pull request #79: add possibility to configure if exec for sftp is disabled
ddg-igh opened a new pull request #79: add possibility to configure if exec for sftp is disabled URL: https://github.com/apache/commons-vfs/pull/79 One of our customer has an interesting sftp server. The server does not allow any further channels to be established after the execution of the exec command (getUid()). So it is not possible to do any filesystem operations, because at first the exec command is executed to test if exec is disabled. Maybe the server things we try to break the security or something else. So my idea to solve this problem is to make execDisabled configurable. This is done here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (COMPRESS-503) "open when actually needed" for MultiReadOnlySeekableByteChannel
[
https://issues.apache.org/jira/browse/COMPRESS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-503:
Affects Version/s: 1.19
> "open when actually needed" for MultiReadOnlySeekableByteChannel
>
>
> Key: COMPRESS-503
> URL: https://issues.apache.org/jira/browse/COMPRESS-503
> Project: Commons Compress
> Issue Type: Improvement
>Affects Versions: 1.19, 1.20
>Reporter: Peter Alfred Lee
>Priority: Minor
> Fix For: 1.21
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I was adding zip64 support for split zip, I encountered a problem :
> When adding testcases in {{Zip64SupportIT}}, I created a split zip with
> 10,000+ split segments. Then I found that I was unable to unzip it because
> there would be too many open files when extracting it. We can oepn the files
> when actually needed and therefore we can successfully extract such split
> zips with great amount of segments.
> I have set a threshold of 20 in {{MultiReadOnlySeekableByteChannel}}. The
> "open when actually needed" procedure will only work when the number of split
> segments is greater than the threshold.
>
> This may be used in ZipArchiveInputStream and ZipFile cause
> {{MultiReadOnlySeekableByteChannel}} is used in them.
>
> Actually this is a pretty rare case cause most split zips would not have too
> many segments. Just thinking about a split zip with 1,000+ segments - it must
> be a nightmare. So I'm not sure if this is needed for
> {{MultiReadOnlySeekableByteChannel.WDYT?}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (COMPRESS-503) "open when actually needed" for MultiReadOnlySeekableByteChannel
[
https://issues.apache.org/jira/browse/COMPRESS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-503:
Fix Version/s: 1.21
> "open when actually needed" for MultiReadOnlySeekableByteChannel
>
>
> Key: COMPRESS-503
> URL: https://issues.apache.org/jira/browse/COMPRESS-503
> Project: Commons Compress
> Issue Type: Improvement
>Affects Versions: 1.20
>Reporter: Peter Alfred Lee
>Priority: Minor
> Fix For: 1.21
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I was adding zip64 support for split zip, I encountered a problem :
> When adding testcases in {{Zip64SupportIT}}, I created a split zip with
> 10,000+ split segments. Then I found that I was unable to unzip it because
> there would be too many open files when extracting it. We can oepn the files
> when actually needed and therefore we can successfully extract such split
> zips with great amount of segments.
> I have set a threshold of 20 in {{MultiReadOnlySeekableByteChannel}}. The
> "open when actually needed" procedure will only work when the number of split
> segments is greater than the threshold.
>
> This may be used in ZipArchiveInputStream and ZipFile cause
> {{MultiReadOnlySeekableByteChannel}} is used in them.
>
> Actually this is a pretty rare case cause most split zips would not have too
> many segments. Just thinking about a split zip with 1,000+ segments - it must
> be a nightmare. So I'm not sure if this is needed for
> {{MultiReadOnlySeekableByteChannel.WDYT?}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (COMPRESS-503) "open when actually needed" for MultiReadOnlySeekableByteChannel
[
https://issues.apache.org/jira/browse/COMPRESS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Bodewig updated COMPRESS-503:
Affects Version/s: 1.20
> "open when actually needed" for MultiReadOnlySeekableByteChannel
>
>
> Key: COMPRESS-503
> URL: https://issues.apache.org/jira/browse/COMPRESS-503
> Project: Commons Compress
> Issue Type: Improvement
>Affects Versions: 1.20
>Reporter: Peter Alfred Lee
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I was adding zip64 support for split zip, I encountered a problem :
> When adding testcases in {{Zip64SupportIT}}, I created a split zip with
> 10,000+ split segments. Then I found that I was unable to unzip it because
> there would be too many open files when extracting it. We can oepn the files
> when actually needed and therefore we can successfully extract such split
> zips with great amount of segments.
> I have set a threshold of 20 in {{MultiReadOnlySeekableByteChannel}}. The
> "open when actually needed" procedure will only work when the number of split
> segments is greater than the threshold.
>
> This may be used in ZipArchiveInputStream and ZipFile cause
> {{MultiReadOnlySeekableByteChannel}} is used in them.
>
> Actually this is a pretty rare case cause most split zips would not have too
> many segments. Just thinking about a split zip with 1,000+ segments - it must
> be a nightmare. So I'm not sure if this is needed for
> {{MultiReadOnlySeekableByteChannel.WDYT?}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (COMPRESS-503) "open when actually needed" for MultiReadOnlySeekableByteChannel
[
https://issues.apache.org/jira/browse/COMPRESS-503?focusedWorklogId=382148=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382148
]
ASF GitHub Bot logged work on COMPRESS-503:
---
Author: ASF GitHub Bot
Created on: 05/Feb/20 08:42
Start Date: 05/Feb/20 08:42
Worklog Time Spent: 10m
Work Description: coveralls commented on issue #92: COMPRESS-503 : "open
when actually needed" for MultiReadOnlySeekableByteChannel
URL: https://github.com/apache/commons-compress/pull/92#issuecomment-582299600
[](https://coveralls.io/builds/28537848)
Coverage decreased (-0.2%) to 86.779% when pulling
**7ffc7a87236be2fd72b663f2735370a2d183494a on
PeterAlfreadLee:COMPRESS-477-improve** into
**151e0fb3fa241887ae3192cc92c572401c6a182b on apache:master**.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382148)
Time Spent: 20m (was: 10m)
> "open when actually needed" for MultiReadOnlySeekableByteChannel
>
>
> Key: COMPRESS-503
> URL: https://issues.apache.org/jira/browse/COMPRESS-503
> Project: Commons Compress
> Issue Type: Improvement
>Reporter: Peter Alfred Lee
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I was adding zip64 support for split zip, I encountered a problem :
> When adding testcases in {{Zip64SupportIT}}, I created a split zip with
> 10,000+ split segments. Then I found that I was unable to unzip it because
> there would be too many open files when extracting it. We can oepn the files
> when actually needed and therefore we can successfully extract such split
> zips with great amount of segments.
> I have set a threshold of 20 in {{MultiReadOnlySeekableByteChannel}}. The
> "open when actually needed" procedure will only work when the number of split
> segments is greater than the threshold.
>
> This may be used in ZipArchiveInputStream and ZipFile cause
> {{MultiReadOnlySeekableByteChannel}} is used in them.
>
> Actually this is a pretty rare case cause most split zips would not have too
> many segments. Just thinking about a split zip with 1,000+ segments - it must
> be a nightmare. So I'm not sure if this is needed for
> {{MultiReadOnlySeekableByteChannel.WDYT?}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] coveralls edited a comment on issue #92: COMPRESS-503 : "open when actually needed" for MultiReadOnlySeekableByteChannel
coveralls edited a comment on issue #92: COMPRESS-503 : "open when actually needed" for MultiReadOnlySeekableByteChannel URL: https://github.com/apache/commons-compress/pull/92#issuecomment-582299600 [](https://coveralls.io/builds/28537848) Coverage decreased (-0.2%) to 86.779% when pulling **7ffc7a87236be2fd72b663f2735370a2d183494a on PeterAlfreadLee:COMPRESS-477-improve** into **151e0fb3fa241887ae3192cc92c572401c6a182b on apache:master**. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (COMPRESS-503) "open when actually needed" for MultiReadOnlySeekableByteChannel
[
https://issues.apache.org/jira/browse/COMPRESS-503?focusedWorklogId=382147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382147
]
ASF GitHub Bot logged work on COMPRESS-503:
---
Author: ASF GitHub Bot
Created on: 05/Feb/20 08:40
Start Date: 05/Feb/20 08:40
Worklog Time Spent: 10m
Work Description: coveralls commented on issue #92: COMPRESS-503 : "open
when actually needed" for MultiReadOnlySeekableByteChannel
URL: https://github.com/apache/commons-compress/pull/92#issuecomment-582299600
[](https://coveralls.io/builds/28537848)
Coverage decreased (-0.2%) to 86.754% when pulling
**7ffc7a87236be2fd72b663f2735370a2d183494a on
PeterAlfreadLee:COMPRESS-477-improve** into
**151e0fb3fa241887ae3192cc92c572401c6a182b on apache:master**.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382147)
Remaining Estimate: 0h
Time Spent: 10m
> "open when actually needed" for MultiReadOnlySeekableByteChannel
>
>
> Key: COMPRESS-503
> URL: https://issues.apache.org/jira/browse/COMPRESS-503
> Project: Commons Compress
> Issue Type: Improvement
>Reporter: Peter Alfred Lee
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When I was adding zip64 support for split zip, I encountered a problem :
> When adding testcases in {{Zip64SupportIT}}, I created a split zip with
> 10,000+ split segments. Then I found that I was unable to unzip it because
> there would be too many open files when extracting it. We can oepn the files
> when actually needed and therefore we can successfully extract such split
> zips with great amount of segments.
> I have set a threshold of 20 in {{MultiReadOnlySeekableByteChannel}}. The
> "open when actually needed" procedure will only work when the number of split
> segments is greater than the threshold.
>
> This may be used in ZipArchiveInputStream and ZipFile cause
> {{MultiReadOnlySeekableByteChannel}} is used in them.
>
> Actually this is a pretty rare case cause most split zips would not have too
> many segments. Just thinking about a split zip with 1,000+ segments - it must
> be a nightmare. So I'm not sure if this is needed for
> {{MultiReadOnlySeekableByteChannel.WDYT?}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] coveralls commented on issue #92: COMPRESS-503 : "open when actually needed" for MultiReadOnlySeekableByteChannel
coveralls commented on issue #92: COMPRESS-503 : "open when actually needed" for MultiReadOnlySeekableByteChannel URL: https://github.com/apache/commons-compress/pull/92#issuecomment-582299600 [](https://coveralls.io/builds/28537848) Coverage decreased (-0.2%) to 86.754% when pulling **7ffc7a87236be2fd72b663f2735370a2d183494a on PeterAlfreadLee:COMPRESS-477-improve** into **151e0fb3fa241887ae3192cc92c572401c6a182b on apache:master**. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (COMPRESS-503) "open when actually needed" for MultiReadOnlySeekableByteChannel
Peter Alfred Lee created COMPRESS-503:
-
Summary: "open when actually needed" for
MultiReadOnlySeekableByteChannel
Key: COMPRESS-503
URL: https://issues.apache.org/jira/browse/COMPRESS-503
Project: Commons Compress
Issue Type: Improvement
Reporter: Peter Alfred Lee
When I was adding zip64 support for split zip, I encountered a problem :
When adding testcases in {{Zip64SupportIT}}, I created a split zip with 10,000+
split segments. Then I found that I was unable to unzip it because there would
be too many open files when extracting it. We can oepn the files when actually
needed and therefore we can successfully extract such split zips with great
amount of segments.
I have set a threshold of 20 in {{MultiReadOnlySeekableByteChannel}}. The "open
when actually needed" procedure will only work when the number of split
segments is greater than the threshold.
This may be used in ZipArchiveInputStream and ZipFile cause
{{MultiReadOnlySeekableByteChannel}} is used in them.
Actually this is a pretty rare case cause most split zips would not have too
many segments. Just thinking about a split zip with 1,000+ segments - it must
be a nightmare. So I'm not sure if this is needed for
{{MultiReadOnlySeekableByteChannel.WDYT?}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] PeterAlfreadLee commented on issue #92: COMPRESS-477 : "open when actually needed" for MultiReadOnlySeekableByteChannel
PeterAlfreadLee commented on issue #92: COMPRESS-477 : "open when actually needed" for MultiReadOnlySeekableByteChannel URL: https://github.com/apache/commons-compress/pull/92#issuecomment-582290001 Sure. I will add a new JIRA issue and modify the name of this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (COMPRESS-477) Support for split zip files
[
https://issues.apache.org/jira/browse/COMPRESS-477?focusedWorklogId=382133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382133
]
ASF GitHub Bot logged work on COMPRESS-477:
---
Author: ASF GitHub Bot
Created on: 05/Feb/20 08:12
Start Date: 05/Feb/20 08:12
Worklog Time Spent: 10m
Work Description: PeterAlfreadLee commented on issue #92: COMPRESS-477 :
"open when actually needed" for MultiReadOnlySeekableByteChannel
URL: https://github.com/apache/commons-compress/pull/92#issuecomment-582290001
Sure. I will add a new JIRA issue and modify the name of this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382133)
Time Spent: 14.5h (was: 14h 20m)
> Support for split zip files
> ---
>
> Key: COMPRESS-477
> URL: https://issues.apache.org/jira/browse/COMPRESS-477
> Project: Commons Compress
> Issue Type: New Feature
> Components: Archivers
>Affects Versions: 1.18
>Reporter: Luís Filipe Nassif
>Priority: Major
> Labels: zip
> Fix For: 1.20
>
> Time Spent: 14.5h
> Remaining Estimate: 0h
>
> It would be very useful to support splitted zip files. I've read
> [https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT] and understood
> that simply concatenating the segments and removing the split signature
> 0x08074b50 from first segment would be sufficient, but it is not that simple
> because compress fails with exception below:
> {code}
> Caused by: java.util.zip.ZipException: archive's ZIP64 end of central
> directory locator is corrupt.
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory64(ZipFile.java:924)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:901)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:621)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:295)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:280)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:236)
> ~[commons-compress-1.18.jar:1.18]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (COMPRESS-477) Support for split zip files
[
https://issues.apache.org/jira/browse/COMPRESS-477?focusedWorklogId=382131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382131
]
ASF GitHub Bot logged work on COMPRESS-477:
---
Author: ASF GitHub Bot
Created on: 05/Feb/20 08:11
Start Date: 05/Feb/20 08:11
Worklog Time Spent: 10m
Work Description: bodewig commented on issue #92: COMPRESS-477 : "open
when actually needed" for MultiReadOnlySeekableByteChannel
URL: https://github.com/apache/commons-compress/pull/92#issuecomment-582289566
@PeterAlfreadLee I think it would be good to track this as a separate JIRA
issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382131)
Time Spent: 14h 20m (was: 14h 10m)
> Support for split zip files
> ---
>
> Key: COMPRESS-477
> URL: https://issues.apache.org/jira/browse/COMPRESS-477
> Project: Commons Compress
> Issue Type: New Feature
> Components: Archivers
>Affects Versions: 1.18
>Reporter: Luís Filipe Nassif
>Priority: Major
> Labels: zip
> Fix For: 1.20
>
> Time Spent: 14h 20m
> Remaining Estimate: 0h
>
> It would be very useful to support splitted zip files. I've read
> [https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT] and understood
> that simply concatenating the segments and removing the split signature
> 0x08074b50 from first segment would be sufficient, but it is not that simple
> because compress fails with exception below:
> {code}
> Caused by: java.util.zip.ZipException: archive's ZIP64 end of central
> directory locator is corrupt.
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory64(ZipFile.java:924)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:901)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:621)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:295)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:280)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:236)
> ~[commons-compress-1.18.jar:1.18]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] bodewig commented on issue #92: COMPRESS-477 : "open when actually needed" for MultiReadOnlySeekableByteChannel
bodewig commented on issue #92: COMPRESS-477 : "open when actually needed" for MultiReadOnlySeekableByteChannel URL: https://github.com/apache/commons-compress/pull/92#issuecomment-582289566 @PeterAlfreadLee I think it would be good to track this as a separate JIRA issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [commons-compress] PeterAlfreadLee opened a new pull request #92: COMPRESS-477 : "open when actually needed" for MultiReadOnlySeekableByteChannel
PeterAlfreadLee opened a new pull request #92: COMPRESS-477 : "open when actually needed" for MultiReadOnlySeekableByteChannel URL: https://github.com/apache/commons-compress/pull/92 Please refer (#90)[https://github.com/apache/commons-compress/pull/90] When I was adding zip64 support for split zip, I encountered a problem : When adding testcases in `Zip64SupportIT`, I created a split zip with 10,000+ split segments. Then I found that I was unable to unzip it because there would be too many open files when extracting it. We can oepn the files when actually needed and therefore we can successfully extract such split zips with great amount of segments. I have set a threshold of 20 in `MultiReadOnlySeekableByteChannel`. The "open when actually needed" procedure will only work when the number of split segments is greater than the threshold. Actually this is a pretty rare case cause most split zips would not have too many segments. So you can decide whether to merge this PR or not. :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (COMPRESS-477) Support for split zip files
[
https://issues.apache.org/jira/browse/COMPRESS-477?focusedWorklogId=382121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382121
]
ASF GitHub Bot logged work on COMPRESS-477:
---
Author: ASF GitHub Bot
Created on: 05/Feb/20 07:59
Start Date: 05/Feb/20 07:59
Worklog Time Spent: 10m
Work Description: PeterAlfreadLee commented on pull request #92:
COMPRESS-477 : "open when actually needed" for MultiReadOnlySeekableByteChannel
URL: https://github.com/apache/commons-compress/pull/92
Please refer (#90)[https://github.com/apache/commons-compress/pull/90]
When I was adding zip64 support for split zip, I encountered a problem :
When adding testcases in `Zip64SupportIT`, I created a split zip with
10,000+ split segments. Then I found that I was unable to unzip it because
there would be too many open files when extracting it. We can oepn the files
when actually needed and therefore we can successfully extract such split zips
with great amount of segments.
I have set a threshold of 20 in `MultiReadOnlySeekableByteChannel`. The
"open when actually needed" procedure will only work when the number of split
segments is greater than the threshold.
Actually this is a pretty rare case cause most split zips would not have too
many segments. So you can decide whether to merge this PR or not. :-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382121)
Time Spent: 14h 10m (was: 14h)
> Support for split zip files
> ---
>
> Key: COMPRESS-477
> URL: https://issues.apache.org/jira/browse/COMPRESS-477
> Project: Commons Compress
> Issue Type: New Feature
> Components: Archivers
>Affects Versions: 1.18
>Reporter: Luís Filipe Nassif
>Priority: Major
> Labels: zip
> Fix For: 1.20
>
> Time Spent: 14h 10m
> Remaining Estimate: 0h
>
> It would be very useful to support splitted zip files. I've read
> [https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT] and understood
> that simply concatenating the segments and removing the split signature
> 0x08074b50 from first segment would be sufficient, but it is not that simple
> because compress fails with exception below:
> {code}
> Caused by: java.util.zip.ZipException: archive's ZIP64 end of central
> directory locator is corrupt.
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory64(ZipFile.java:924)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:901)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:621)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:295)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:280)
> ~[commons-compress-1.18.jar:1.18]
> at
> org.apache.commons.compress.archivers.zip.ZipFile.(ZipFile.java:236)
> ~[commons-compress-1.18.jar:1.18]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
