[jira] [Comment Edited] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743347#comment-17743347 ] Quan Li edited comment on HADOOP-18291 at 7/15/23 5:57 AM: --- the tests are failing in our internal, can't follow the code, tons of review comments, not sure if reviewers wrote code via review this much comments. [~mthakur] [~mehakmeet] [~ayushsaxena] [~hexiaoqiao] [~inigoiri] Can someone fix/revert this ticket -> let reviewer fix it via review -> still break -> addendum very tough backporting such was (Author: quanli): the tests are failing in our internal, can't follow the code, tons of review comments, not sure if reviewers wrote code via review this much comments. [~mthakur] [~mehakmeet] [~ayushsaxena] [~hexiaoqiao] [~inigoiri] Can someone fix/revert this > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743347#comment-17743347 ] Quan Li commented on HADOOP-18291: -- the tests are failing in our internal, can't follow the code, tons of review comments, not sure if reviewers wrote code via review this much comments. [~mthakur] [~mehakmeet] [~ayushsaxena] [~hexiaoqiao] [~inigoiri] Can someone fix/revert this > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18806) Document missing property (ipc.server.read.threadpool.size) in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18806: Labels: pull-request-available (was: ) > Document missing property (ipc.server.read.threadpool.size) in > core-default.xml > --- > > Key: HADOOP-18806 > URL: https://issues.apache.org/jira/browse/HADOOP-18806 > Project: Hadoop Common > Issue Type: Improvement >Reporter: YUBI LEE >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18806) Document missing property (ipc.server.read.threadpool.size) in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743346#comment-17743346 ] ASF GitHub Bot commented on HADOOP-18806: - eubnara opened a new pull request, #5844: URL: https://github.com/apache/hadoop/pull/5844 ### Description of PR I think `ipc.server.read.threadpool.size` is quite important configuration tuning point. But there is no mention about it in core-default.xml. There are some articles about hadoop tuning configuration with it. (e.g. https://support.huawei.com/enterprise/en/knowledge/EKB1100015760) ### How was this patch tested? It is just changed documentation. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? > Document missing property (ipc.server.read.threadpool.size) in > core-default.xml > --- > > Key: HADOOP-18806 > URL: https://issues.apache.org/jira/browse/HADOOP-18806 > Project: Hadoop Common > Issue Type: Improvement >Reporter: YUBI LEE >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] eubnara opened a new pull request, #5844: HADOOP-18806. Document missing property (ipc.server.read.threadpool.size) in core-default.xml
eubnara opened a new pull request, #5844: URL: https://github.com/apache/hadoop/pull/5844 ### Description of PR I think `ipc.server.read.threadpool.size` is quite important configuration tuning point. But there is no mention about it in core-default.xml. There are some articles about hadoop tuning configuration with it. (e.g. https://support.huawei.com/enterprise/en/knowledge/EKB1100015760) ### How was this patch tested? It is just changed documentation. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18806) Document missing property (ipc.server.read.threadpool.size) in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YUBI LEE updated HADOOP-18806: -- Summary: Document missing property (ipc.server.read.threadpool.size) in core-default.xml (was: Document missing property (ipc.server.read.threadpool.size) in core-site.xml) > Document missing property (ipc.server.read.threadpool.size) in > core-default.xml > --- > > Key: HADOOP-18806 > URL: https://issues.apache.org/jira/browse/HADOOP-18806 > Project: Hadoop Common > Issue Type: Improvement >Reporter: YUBI LEE >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18806) Document missing property (ipc.server.read.threadpool.size) in core-site.xml
YUBI LEE created HADOOP-18806: - Summary: Document missing property (ipc.server.read.threadpool.size) in core-site.xml Key: HADOOP-18806 URL: https://issues.apache.org/jira/browse/HADOOP-18806 Project: Hadoop Common Issue Type: Improvement Reporter: YUBI LEE -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data
[
https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743344#comment-17743344
]
Viraj Jasani commented on HADOOP-18805:
---
sorry Steve, i was not aware you already created this Jira, i created addendum
for letting LRU test depend on small file rather than large one:
[https://github.com/apache/hadoop/pull/5843]
{quote}also, and this is very, very important, they can't validate the data
{quote}
i was about to create a sub-task for this as i am planning to refactor Entry to
it's own class and have the contents of the linked list data tested in UT
(discussed with Mehakmeet in the earlier part of the review). maybe i can do
the work as part of this Jira.
are you fine with the above addendum PR taking care of using small file in the
test (so that we don't need to put the test under -scale) and this Jira being
used for refactoring Entry and allowing a UT to test the contents of the linked
list?
> s3a large file prefetch tests are too slow, don't validate data
> ---
>
> Key: HADOOP-18805
> URL: https://issues.apache.org/jira/browse/HADOOP-18805
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.9
>Reporter: Steve Loughran
>Priority: Major
>
> the large file prefetch tests (including LRU cache eviction) are really slow.
> moving under -scale may hide the problem for most runs, but they are still
> too slow, can time out, etc etc.
> also, and this is very, very important, they can't validate the data.
> Better:
> * test on smaller files by setting a very small block size (1k bytes or less)
> just to force paged reads of a small 16k file.
> * with known contents to the values of all forms of read can be validated
> * maybe the LRU tests can work with a fake remote object which can then be
> used in a unit test
> * extend one of the huge file tests to read from there -including s3-CSE
> encryption coverage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani reassigned HADOOP-18805: - Assignee: (was: Viraj Jasani) > s3a large file prefetch tests are too slow, don't validate data > --- > > Key: HADOOP-18805 > URL: https://issues.apache.org/jira/browse/HADOOP-18805 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.9 >Reporter: Steve Loughran >Priority: Major > > the large file prefetch tests (including LRU cache eviction) are really slow. > moving under -scale may hide the problem for most runs, but they are still > too slow, can time out, etc etc. > also, and this is very, very important, they can't validate the data. > Better: > * test on smaller files by setting a very small block size (1k bytes or less) > just to force paged reads of a small 16k file. > * with known contents to the values of all forms of read can be validated > * maybe the LRU tests can work with a fake remote object which can then be > used in a unit test > * extend one of the huge file tests to read from there -including s3-CSE > encryption coverage. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data
[ https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani reassigned HADOOP-18805: - Assignee: Viraj Jasani > s3a large file prefetch tests are too slow, don't validate data > --- > > Key: HADOOP-18805 > URL: https://issues.apache.org/jira/browse/HADOOP-18805 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.9 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > > the large file prefetch tests (including LRU cache eviction) are really slow. > moving under -scale may hide the problem for most runs, but they are still > too slow, can time out, etc etc. > also, and this is very, very important, they can't validate the data. > Better: > * test on smaller files by setting a very small block size (1k bytes or less) > just to force paged reads of a small 16k file. > * with known contents to the values of all forms of read can be validated > * maybe the LRU tests can work with a fake remote object which can then be > used in a unit test > * extend one of the huge file tests to read from there -including s3-CSE > encryption coverage. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743334#comment-17743334 ] ASF GitHub Bot commented on HADOOP-18291: - virajjasani commented on PR #5843: URL: https://github.com/apache/hadoop/pull/5843#issuecomment-1636603969 not sure, what is going wrong with jenkins env https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5843/1/console ``` Error when executing cleanup post condition: org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing Perhaps you forgot to surround the code with a step that provides this, such as: node at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:265) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:299) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124) ``` ``` Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to hadoop2 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:1000) at hudson.FilePath.act(FilePath.java:1194) at hudson.FilePath.act(FilePath.java:1183) at hudson.FilePath.mkdirs(FilePath.java:1374) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:844) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1296) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) java.nio.file.FileSystemException: /home/jenkins/jenkins-home/workspace/hadoop-multibranch: Read-only file system at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) at java.base/java.nio.file.Files.createDirectory(Files.java:690) at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:797) ``` > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5843: HADOOP-18291. (ADDENDUM) S3A prefetch LRU test to work with small file
virajjasani commented on PR #5843: URL: https://github.com/apache/hadoop/pull/5843#issuecomment-1636603969 not sure, what is going wrong with jenkins env https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5843/1/console ``` Error when executing cleanup post condition: org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing Perhaps you forgot to surround the code with a step that provides this, such as: node at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:265) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:299) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124) ``` ``` Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to hadoop2 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:1000) at hudson.FilePath.act(FilePath.java:1194) at hudson.FilePath.act(FilePath.java:1183) at hudson.FilePath.mkdirs(FilePath.java:1374) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:844) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1296) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) java.nio.file.FileSystemException: /home/jenkins/jenkins-home/workspace/hadoop-multibranch: Read-only file system at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) at java.base/java.nio.file.Files.createDirectory(Files.java:690) at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:797) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1774#comment-1774 ] ASF GitHub Bot commented on HADOOP-18184: - virajjasani commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636601526 > the solution here shouldn't be "add a bigger timeout" it should be "make these tests faster by working with smaller files and smaller blocks" PR #5843 > s3a prefetching stream to support unbuffer() > > > Key: HADOOP-18184 > URL: https://issues.apache.org/jira/browse/HADOOP-18184 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > Apache Impala uses unbuffer() to free up all client side resources held by a > stream, so allowing it to have a map of available (path -> stream) objects, > retained across queries. > This saves on having to reopen the files, with the cost of HEAD checks etc. > S3AInputStream just closes its http connection. here there is a lot more > state to discard, but all memory and file storage must be freed. > until this done, ITestS3AContractUnbuffer must skip when the prefetch stream > is used. > its notable that the other tests don't fail, even though the stream doesn't > implement the interface; the graceful degradation handles that. it should > fail if the test xml resource says the stream does it, but that the stream > capabilities say it doesn't. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
virajjasani commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636601526 > the solution here shouldn't be "add a bigger timeout" it should be "make these tests faster by working with smaller files and smaller blocks" PR #5843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743332#comment-17743332 ] ASF GitHub Bot commented on HADOOP-18291: - virajjasani commented on PR #5754: URL: https://github.com/apache/hadoop/pull/5754#issuecomment-1636598987 created addendum PR for dealing with small file #5843 > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on PR #5754: URL: https://github.com/apache/hadoop/pull/5754#issuecomment-1636598987 created addendum PR for dealing with small file #5843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743331#comment-17743331 ] ASF GitHub Bot commented on HADOOP-18291: - virajjasani opened a new pull request, #5843: URL: https://github.com/apache/hadoop/pull/5843 (no comment) > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani opened a new pull request, #5843: HADOOP-18291. (ADDENDUM) S3A prefetch LRU test to work with small file
virajjasani opened a new pull request, #5843: URL: https://github.com/apache/hadoop/pull/5843 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[
https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743280#comment-17743280
]
ASF GitHub Bot commented on HADOOP-18184:
-
hadoop-yetus commented on PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636399060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 19 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 19s | | trunk passed |
| +1 :green_heart: | compile | 18m 30s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 16m 59s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 35s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 1s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 17m 54s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 17m 54s | | the patch passed |
| +1 :green_heart: | compile | 16m 52s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 35s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 24 new + 5 unchanged - 0 fixed = 29 total (was 5)
|
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javadoc | 0m 42s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09
with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 generated 4
new + 0 unchanged - 0 fixed = 4 total (was 0) |
| +1 :green_heart: | spotbugs | 4m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 50s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 2m 44s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 260m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.s3a.prefetch.TestS3ARemoteInputStream |
| | hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
[GitHub] [hadoop] hadoop-yetus commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
hadoop-yetus commented on PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636399060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 19 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 19s | | trunk passed |
| +1 :green_heart: | compile | 18m 30s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 16m 59s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 35s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 1s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 17m 54s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 17m 54s | | the patch passed |
| +1 :green_heart: | compile | 16m 52s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 35s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 24 new + 5 unchanged - 0 fixed = 29 total (was 5)
|
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javadoc | 0m 42s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09
with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 generated 4
new + 0 unchanged - 0 fixed = 4 total (was 0) |
| +1 :green_heart: | spotbugs | 4m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 50s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 2m 44s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 260m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.s3a.prefetch.TestS3ARemoteInputStream |
| | hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5832 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit
[jira] [Commented] (HADOOP-18795) s3a DelegationToken plugin to expand return type of deploy/binding
[
https://issues.apache.org/jira/browse/HADOOP-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743279#comment-17743279
]
ASF GitHub Bot commented on HADOOP-18795:
-
hadoop-yetus commented on PR #5821:
URL: https://github.com/apache/hadoop/pull/5821#issuecomment-1636368868
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javadoc | 0m 27s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5821/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09
with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 41s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 131m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5821/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5821 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 12372674a5ba 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62a3eea77c8cc41e419f707b1e061abba1355362 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5821: HADOOP-18795. S3A DelegationToken plugin to expand return type of binding
hadoop-yetus commented on PR #5821:
URL: https://github.com/apache/hadoop/pull/5821#issuecomment-1636368868
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javadoc | 0m 27s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5821/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09
with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 41s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 131m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5821/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5821 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 12372674a5ba 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62a3eea77c8cc41e419f707b1e061abba1355362 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5821/2/testReport/ |
| Max. process+thread count | 557 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console
[jira] [Commented] (HADOOP-18801) Delete path directly when it can not be parsed in trash
[
https://issues.apache.org/jira/browse/HADOOP-18801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743271#comment-17743271
]
ASF GitHub Bot commented on HADOOP-18801:
-
hadoop-yetus commented on PR #5744:
URL: https://github.com/apache/hadoop/pull/5744#issuecomment-1636344146
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 9s | | trunk passed |
| +1 :green_heart: | compile | 17m 18s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 16m 12s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 16m 34s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 34s | | the patch passed |
| +1 :green_heart: | compile | 16m 8s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 20s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not
generate ASF License warnings. |
| | | 225m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5744/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5744 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 3216146ba904 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cf0f71640cdf47c4aee76a9fe56f37f1a87d6807 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5744/13/testReport/ |
| Max. process+thread count | 1263 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5744/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[GitHub] [hadoop] hadoop-yetus commented on pull request #5744: HADOOP-18801. Delete path directly when it can not be parsed in trash.
hadoop-yetus commented on PR #5744:
URL: https://github.com/apache/hadoop/pull/5744#issuecomment-1636344146
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 9s | | trunk passed |
| +1 :green_heart: | compile | 17m 18s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 16m 12s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 16m 34s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 34s | | the patch passed |
| +1 :green_heart: | compile | 16m 8s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 20s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not
generate ASF License warnings. |
| | | 225m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5744/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5744 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 3216146ba904 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cf0f71640cdf47c4aee76a9fe56f37f1a87d6807 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5744/13/testReport/ |
| Max. process+thread count | 1263 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5744/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
[GitHub] [hadoop] hadoop-yetus commented on pull request #5842: HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota
hadoop-yetus commented on PR #5842:
URL: https://github.com/apache/hadoop/pull/5842#issuecomment-1636259839
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 60m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 257m 24s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5842/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 425m 43s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5842/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8f17b1d9858f 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f1c8080eac4d92f7cc1992b9034b37f98280a305 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5842/1/testReport/ |
| Max. process+thread count | 2194 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5842/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus
[jira] [Created] (HADOOP-18805) s3a large file prefetch tests are too slow, don't validate data
Steve Loughran created HADOOP-18805: --- Summary: s3a large file prefetch tests are too slow, don't validate data Key: HADOOP-18805 URL: https://issues.apache.org/jira/browse/HADOOP-18805 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3, test Affects Versions: 3.3.9 Reporter: Steve Loughran the large file prefetch tests (including LRU cache eviction) are really slow. moving under -scale may hide the problem for most runs, but they are still too slow, can time out, etc etc. also, and this is very, very important, they can't validate the data. Better: * test on smaller files by setting a very small block size (1k bytes or less) just to force paged reads of a small 16k file. * with known contents to the values of all forms of read can be validated * maybe the LRU tests can work with a fake remote object which can then be used in a unit test * extend one of the huge file tests to read from there -including s3-CSE encryption coverage. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18291: Fix Version/s: 3.3.9 Resolution: Fixed Status: Resolved (was: Patch Available) > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5823: HDFS-17075. Reconfig disk balancer parameters for datanode
hadoop-yetus commented on PR #5823:
URL: https://github.com/apache/hadoop/pull/5823#issuecomment-1636170310
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 2s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5823/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 79 unchanged -
0 fixed = 80 total (was 79) |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 214m 25s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 357m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5823/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5823 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1dc008b0d422 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 55ac41ff89076ed9aedc24c981a861e5c64e5aec |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5823/3/testReport/ |
| Max. process+thread count | 3505 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5823/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743235#comment-17743235 ] ASF GitHub Bot commented on HADOOP-18291: - virajjasani commented on PR #5754: URL: https://github.com/apache/hadoop/pull/5754#issuecomment-1636141106 i see, i was thinking, perhaps small file with prefetch of multiple blocks might be too small for the test? but on the other hand, i agree that we don't need such long running test either, let me at least remove "3" and "4" from block size array and have the test time duration reduce by half for now in an addendum? > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on PR #5754: URL: https://github.com/apache/hadoop/pull/5754#issuecomment-1636141106 i see, i was thinking, perhaps small file with prefetch of multiple blocks might be too small for the test? but on the other hand, i agree that we don't need such long running test either, let me at least remove "3" and "4" from block size array and have the test time duration reduce by half for now in an addendum? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18384) ITestS3AFileSystemStatistic failure in prefetch feature branch
[
https://issues.apache.org/jira/browse/HADOOP-18384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743232#comment-17743232
]
Steve Loughran commented on HADOOP-18384:
-
not seen for a while
> ITestS3AFileSystemStatistic failure in prefetch feature branch
> --
>
> Key: HADOOP-18384
> URL: https://issues.apache.org/jira/browse/HADOOP-18384
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Samrat Deb
>Priority: Minor
>
> testing the rebased prefetch feature branch; got a failure in
> ITestS3AFileSystemStatistic
>
> {code}
> tics.ITestS3AFileSystemStatistic
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.489
> s <<< FAILURE! - in
> org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
> [ERROR]
> testBytesReadWithStream(org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic)
> Time elapsed: 1.489 s <<< FAILURE!
> java.lang.AssertionError: Mismatch in number of FS bytes read by InputStreams
> expected:<2048> but was:<69537130>
> at
> org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic.testBytesReadWithStream(ITestS3AFileSystemStatistic.java:72)
> {code}
> that;s 64MB + ~237 kb, the kind of values you would get from prefetching
> but, prefetch was disabled in this test run.
> maybe its just the fs stats aren't being reset between test cases
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743230#comment-17743230 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636115561 tested, s3 london, with `-Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale` and no VPN in the way. This is getting back to as slow as it used to be -and so needs work. All the landasat tests are going to be long-haul for most people; the existing hugefile tests should be extended to do the reading on their files which are (a) on the chosen aws region and (b) let you control the filesize ``` [INFO] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 29:27 min (Wall Clock) [INFO] Finished at: 2023-07-14T17:34:02+01:00 [INFO] [WARNING] ``` > s3a prefetching stream to support unbuffer() > > > Key: HADOOP-18184 > URL: https://issues.apache.org/jira/browse/HADOOP-18184 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > Apache Impala uses unbuffer() to free up all client side resources held by a > stream, so allowing it to have a map of available (path -> stream) objects, > retained across queries. > This saves on having to reopen the files, with the cost of HEAD checks etc. > S3AInputStream just closes its http connection. here there is a lot more > state to discard, but all memory and file storage must be freed. > until this done, ITestS3AContractUnbuffer must skip when the prefetch stream > is used. > its notable that the other tests don't fail, even though the stream doesn't > implement the interface; the graceful degradation handles that. it should > fail if the test xml resource says the stream does it, but that the stream > capabilities say it doesn't. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636115561 tested, s3 london, with `-Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale` and no VPN in the way. This is getting back to as slow as it used to be -and so needs work. All the landasat tests are going to be long-haul for most people; the existing hugefile tests should be extended to do the reading on their files which are (a) on the chosen aws region and (b) let you control the filesize ``` [INFO] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 29:27 min (Wall Clock) [INFO] Finished at: 2023-07-14T17:34:02+01:00 [INFO] [WARNING] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743228#comment-17743228 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636111884 timeout in lru tests ``` [ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 843.577 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3APrefetchingLruEviction [ERROR] testSeeksWithLruEviction[max-blocks-1](org.apache.hadoop.fs.s3a.ITestS3APrefetchingLruEviction) Time elapsed: 600.017 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 60 milliseconds at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:999) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at org.apache.hadoop.fs.s3a.ITestS3APrefetchingLruEviction.testSeeksWithLruEviction(ITestS3APrefetchingLruEviction.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) ``` issue here is having all the different bulk reads in the same test case; if it takes too long (> 10 minutes!) then it fails. the solution here shouldn't be "add a bigger timeout" it should be "make these tests faster by working with smaller files and smaller blocks" > s3a prefetching stream to support unbuffer() > > > Key: HADOOP-18184 > URL: https://issues.apache.org/jira/browse/HADOOP-18184 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > Apache Impala uses unbuffer() to free up all client side resources held by a > stream, so allowing it to have a map of available (path -> stream) objects, > retained across queries. > This saves on having to reopen the files, with the cost of HEAD checks etc. > S3AInputStream just closes its http connection. here there is a lot more > state to discard, but all memory and file storage must be freed. > until this done, ITestS3AContractUnbuffer must skip when the prefetch stream > is used. > its notable that the other tests don't fail, even though the stream doesn't > implement the interface; the graceful degradation handles that. it should > fail if the test xml resource says the stream does it, but that the stream > capabilities say it doesn't. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636111884 timeout in lru tests ``` [ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 843.577 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3APrefetchingLruEviction [ERROR] testSeeksWithLruEviction[max-blocks-1](org.apache.hadoop.fs.s3a.ITestS3APrefetchingLruEviction) Time elapsed: 600.017 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 60 milliseconds at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:999) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at org.apache.hadoop.fs.s3a.ITestS3APrefetchingLruEviction.testSeeksWithLruEviction(ITestS3APrefetchingLruEviction.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) ``` issue here is having all the different bulk reads in the same test case; if it takes too long (> 10 minutes!) then it fails. the solution here shouldn't be "add a bigger timeout" it should be "make these tests faster by working with smaller files and smaller blocks" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5816: YARN-11525. [Federation] Router CLI Supports Save the SubClusterPolicyConfiguration Of Queues.
hadoop-yetus commented on PR #5816:
URL: https://github.com/apache/hadoop/pull/5816#issuecomment-1636101720
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 50s | | trunk passed |
| +1 :green_heart: | compile | 7m 46s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 7m 14s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 52s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 6m 28s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 45s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 4s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 0s | | the patch passed |
| +1 :green_heart: | compile | 7m 2s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 7m 2s | | the patch passed |
| +1 :green_heart: | javac | 7m 2s | | the patch passed |
| +1 :green_heart: | compile | 7m 14s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 7m 14s | | the patch passed |
| +1 :green_heart: | javac | 7m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 52s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5816/17/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 65 unchanged -
0 fixed = 71 total (was 65) |
| +1 :green_heart: | mvnsite | 6m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 7s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 5m 53s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 39s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 33m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 20s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 56s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 41s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 100m 53s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 50s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 53s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | unit | 0m 45s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 369m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743212#comment-17743212 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636074815 Yes, this is a lot more than just unbuffer, but its the first time i've really had the code in the IDE with me writing tests to use IOStats, context iostats, waiting for tests to finish etc. I have more to do which I will followup on different jiras. key: actually support small block memory caching so you can use the stream without any disk use. needed to switch to this everywhere. > s3a prefetching stream to support unbuffer() > > > Key: HADOOP-18184 > URL: https://issues.apache.org/jira/browse/HADOOP-18184 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > Apache Impala uses unbuffer() to free up all client side resources held by a > stream, so allowing it to have a map of available (path -> stream) objects, > retained across queries. > This saves on having to reopen the files, with the cost of HEAD checks etc. > S3AInputStream just closes its http connection. here there is a lot more > state to discard, but all memory and file storage must be freed. > until this done, ITestS3AContractUnbuffer must skip when the prefetch stream > is used. > its notable that the other tests don't fail, even though the stream doesn't > implement the interface; the graceful degradation handles that. it should > fail if the test xml resource says the stream does it, but that the stream > capabilities say it doesn't. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636074815 Yes, this is a lot more than just unbuffer, but its the first time i've really had the code in the IDE with me writing tests to use IOStats, context iostats, waiting for tests to finish etc. I have more to do which I will followup on different jiras. key: actually support small block memory caching so you can use the stream without any disk use. needed to switch to this everywhere. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743209#comment-17743209 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636071189 HADOOP-18184. S3A prefetch unbuffer * Lots of statistic collection with use in tests. * s3a prefetch tests all moved to prefetch. package * and split into caching stream and large files tests * large files and LRU are scale * and testRandomReadLargeFile uses small block size to reduce read overhead * new hadoop common org.apache.hadoop.test.Sizes sizes class with predefined sizes (from azure; not moved existing code to it yet) Overall, the prefetch reads of the large files are slow; while it's critical to test multi-block files, we don't need to work on the landsat csv file. better: one of the huge tests uses it, with a small block size of 1 MB to force lots of work. > s3a prefetching stream to support unbuffer() > > > Key: HADOOP-18184 > URL: https://issues.apache.org/jira/browse/HADOOP-18184 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > Apache Impala uses unbuffer() to free up all client side resources held by a > stream, so allowing it to have a map of available (path -> stream) objects, > retained across queries. > This saves on having to reopen the files, with the cost of HEAD checks etc. > S3AInputStream just closes its http connection. here there is a lot more > state to discard, but all memory and file storage must be freed. > until this done, ITestS3AContractUnbuffer must skip when the prefetch stream > is used. > its notable that the other tests don't fail, even though the stream doesn't > implement the interface; the graceful degradation handles that. it should > fail if the test xml resource says the stream does it, but that the stream > capabilities say it doesn't. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1636071189 HADOOP-18184. S3A prefetch unbuffer * Lots of statistic collection with use in tests. * s3a prefetch tests all moved to prefetch. package * and split into caching stream and large files tests * large files and LRU are scale * and testRandomReadLargeFile uses small block size to reduce read overhead * new hadoop common org.apache.hadoop.test.Sizes sizes class with predefined sizes (from azure; not moved existing code to it yet) Overall, the prefetch reads of the large files are slow; while it's critical to test multi-block files, we don't need to work on the landsat csv file. better: one of the huge tests uses it, with a small block size of 1 MB to force lots of work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743208#comment-17743208 ] ASF GitHub Bot commented on HADOOP-18291: - steveloughran commented on PR #5754: URL: https://github.com/apache/hadoop/pull/5754#issuecomment-1636070773 merged, though now i'm using it that new test is way too slow. in my rebased unbuffered pr I have moved it to -Dscale, but really we can just set the block size down to something minimal and then work with a small file > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
steveloughran commented on PR #5754: URL: https://github.com/apache/hadoop/pull/5754#issuecomment-1636070773 merged, though now i'm using it that new test is way too slow. in my rebased unbuffered pr I have moved it to -Dscale, but really we can just set the block size down to something minimal and then work with a small file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18801) Delete path directly when it can not be parsed in trash
[
https://issues.apache.org/jira/browse/HADOOP-18801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743205#comment-17743205
]
ASF GitHub Bot commented on HADOOP-18801:
-
ayushtkn commented on code in PR #5744:
URL: https://github.com/apache/hadoop/pull/5744#discussion_r1263885805
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java:
##
@@ -374,8 +382,14 @@ private void deleteCheckpoint(Path trashRoot, boolean
deleteImmediately)
try {
time = getTimeFromCheckpoint(name);
} catch (ParseException e) {
-LOG.warn("Unexpected item in trash: "+dir+". Ignoring.");
-continue;
+if (cleanNonCheckpointUnderTrashRoot) {
+ fs.delete(path, true);
+ LOG.warn("Unexpected item in trash: " + dir + ". Force to delete
it.");
Review Comment:
can we change to
```
LOG.warn("Unexpected item in trash: " + dir + ". Deleting.");
```
##
hadoop-common-project/hadoop-common/src/main/resources/core-default.xml:
##
@@ -974,6 +974,14 @@
+
+ fs.trash.clean.trashroot.enable
+ false
+ Whether clean some directories or files
+at home of Trash which are not under checkpoint directory or not.
+
Review Comment:
some error in language, can we change it something like
```
Whether to delete directories and files in Trash home which are not under
checkpoint directory
```
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java:
##
@@ -786,6 +787,55 @@ public void testTrashEmptier() throws Exception {
emptierThread.join();
}
+ /**
+ * Test trash emptier can whether delete non-checkpoint dir or not.
Review Comment:
typo: can & whether don't go together
```
Test trash emptier can delete non-checkpoint dir or not.
```
> Delete path directly when it can not be parsed in trash
> ---
>
> Key: HADOOP-18801
> URL: https://issues.apache.org/jira/browse/HADOOP-18801
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> If we move path to trash dir directly rather than use delete API or rm
> command, when
> invoke deleteCheckpoint method, it will catch ParseException and ignore
> deleting the path. It will never be deleted, so we should do something to
> prevent or monitor it.
> Some logs are listed below.
>
> {code:java}
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Unexpected item in trash:
> /user/de_eight/.Trash/college_geek_job_recall_als_modelres_5_2_6.del.
> Ignoring.
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Unexpected item in trash:
> /user/de_eight/.Trash/college_geek_addf_vector. Ignoring.
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a diff in pull request #5744: HADOOP-18801. Delete path directly when it can not be parsed in trash.
ayushtkn commented on code in PR #5744:
URL: https://github.com/apache/hadoop/pull/5744#discussion_r1263885805
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java:
##
@@ -374,8 +382,14 @@ private void deleteCheckpoint(Path trashRoot, boolean
deleteImmediately)
try {
time = getTimeFromCheckpoint(name);
} catch (ParseException e) {
-LOG.warn("Unexpected item in trash: "+dir+". Ignoring.");
-continue;
+if (cleanNonCheckpointUnderTrashRoot) {
+ fs.delete(path, true);
+ LOG.warn("Unexpected item in trash: " + dir + ". Force to delete
it.");
Review Comment:
can we change to
```
LOG.warn("Unexpected item in trash: " + dir + ". Deleting.");
```
##
hadoop-common-project/hadoop-common/src/main/resources/core-default.xml:
##
@@ -974,6 +974,14 @@
+
+ fs.trash.clean.trashroot.enable
+ false
+ Whether clean some directories or files
+at home of Trash which are not under checkpoint directory or not.
+
Review Comment:
some error in language, can we change it something like
```
Whether to delete directories and files in Trash home which are not under
checkpoint directory
```
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java:
##
@@ -786,6 +787,55 @@ public void testTrashEmptier() throws Exception {
emptierThread.join();
}
+ /**
+ * Test trash emptier can whether delete non-checkpoint dir or not.
Review Comment:
typo: can & whether don't go together
```
Test trash emptier can delete non-checkpoint dir or not.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #5809: HDFS-17068. Datanode should record last directory scan time.
ayushtkn merged PR #5809: URL: https://github.com/apache/hadoop/pull/5809 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhtttylz commented on pull request #5836: HDFS-17083. Support getErasureCodeCodecs API in WebHDFS
zhtttylz commented on PR #5836: URL: https://github.com/apache/hadoop/pull/5836#issuecomment-1636019383 @ayushtkn @slfan1989 Thank you for your assistance in reviewing the code! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18804) Out of Memory when mistakenly set io.file.buffer.size to a large number
ConfX created HADOOP-18804:
--
Summary: Out of Memory when mistakenly set io.file.buffer.size to
a large number
Key: HADOOP-18804
URL: https://issues.apache.org/jira/browse/HADOOP-18804
Project: Hadoop Common
Issue Type: Bug
Components: common, fs
Reporter: ConfX
h2. What happened:
When setting io.file.buffer.size to a large number,
BufferedIOStatisticsOutputStream in Hcommon throws an out-of-memory exception
due to inappropriate checking and handling.
The config is used to initialize a file system by passing it as one of the
parameters bufferSize.
h2. Buggy code:
In RawLocalFileSystem.java
{code:java}
private FSDataOutputStream create(Path f, boolean overwrite,
boolean createParent, int bufferSize, short replication, long blockSize,
Progressable progress, FsPermission permission) throws IOException {
...
return new FSDataOutputStream(new BufferedIOStatisticsOutputStream(
createOutputStreamWithMode(f, false, permission), bufferSize, true), <<---
creates a BufferedIOStatisticsOutputStream with bufferSize, often set to config
io.file.buffer.size
statistics);
} {code}
In BufferedIOStatisticsOutputStream.java:
{code:java}
public class BufferedIOStatisticsOutputStream extends BufferedOutputStream
implements IOStatisticsSource, Syncable, StreamCapabilities {
...
public BufferedIOStatisticsOutputStream(
final OutputStream out,
final int size,
final boolean downgradeSyncable) {
super(out, size); <<--- init the BufferedOutputStream with a huge buffer
size
...
}{code}
h2. StackTrace:
{code:java}
java.lang.OutOfMemoryError: Java heap space
at
java.base/java.io.BufferedOutputStream.(BufferedOutputStream.java:75)
at
org.apache.hadoop.fs.statistics.BufferedIOStatisticsOutputStream.(BufferedIOStatisticsOutputSt
ream.java:78)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:428)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:413)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
at
org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:183)
at
org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:152)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContract
RenameTest.java:335)
...{code}
h2. Reproduce:
(1) Set io.file.buffer.size to a large value, e.g., 2112001717
(2) Run a simple test that exercises this parameter, e.g.
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename#testRenameFileUnderFile
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18802) Collision of config key name fs.viewfs.mounttable.default.name.key to other keys that specify the entry point to mount tables
[
https://issues.apache.org/jira/browse/HADOOP-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18802:
---
Description:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of InodeTree, the tree attempts to get all the mount table
entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here mountTablePrefix="fs.viewfs.mounttable.default.". However, it just so
happens that the name of the configuration users use to specify the default
mount table is fs.viewfs.mounttable.default.name.key. Thus, if a user specifies
the default mount table and uses InodeTree the name.key would be falsely parsed
as the entry point to one of the mount tables, which would cause InodeTree to
throw an exception since name.key is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set fs.viewfs.mounttable.default.name.key to default
(2) Run test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
You can use the reproduce.sh in the attachment to easily reproduce the bug.
We are happy to provide a patch if this issue is confirmed.
was:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of InodeTree, the tree attempts to get all the mount table
entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here mountTablePrefix="fs.viewfs.mounttable.default.". However, it just so
happens that the name of the configuration users use to specify the default
mount table is fs.viewfs.mounttable.default.name.key. Thus, if a user specifies
the default mount table and uses InodeTree the name.key would be falsely parsed
as the entry point to one of the mount tables, which would cause InodeTree to
throw an exception since name.key is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set fs.viewfs.mounttable.default.name.key to default
(2) Run test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
> Collision of config key name fs.viewfs.mounttable.default.name.key to other
> keys that specify the entry point to mount tables
> -
>
> Key: HADOOP-18802
> URL: https://issues.apache.org/jira/browse/HADOOP-18802
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, conf, fs
>Reporter: ConfX
>Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened:
> When manually set
[jira] [Updated] (HADOOP-18803) ClassCastException in test TestRPC#testWrappedStopProxy
[
https://issues.apache.org/jira/browse/HADOOP-18803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18803:
---
Description:
h2. What happened:
In HCommon, test TestRPC#testWrappedStopProxy tries to cast
org.apache.hadoop.ipc.WritableRpcEngine$Invoker to
org.apache.hadoop.ipc.TestRPC$StoppedInvocationHandler and causes
ClassCastException.
h2. Buggy code:
In TestRPC.java:
{code:java}
@Test
public void testWrappedStopProxy() throws IOException {
StoppedProtocol wrappedProxy = RPC.getProxy(StoppedProtocol.class,
StoppedProtocol.versionID, null, conf);
StoppedInvocationHandler invocationHandler = (StoppedInvocationHandler) //
<--- Here causes ClassCastException
Proxy.getInvocationHandler(wrappedProxy);
...
} {code}
h2. How to reproduce:
(1) Just directly run test TestRPC#testWrappedStopProxy.
You can use the reproduce.sh in the attachment to easily reproduce the bug.
We are happy to provide a patch if this issue is confirmed.
was:
h2. What happened:
In HCommon, test TestRPC#testWrappedStopProxy tries to cast
org.apache.hadoop.ipc.WritableRpcEngine$Invoker to
org.apache.hadoop.ipc.TestRPC$StoppedInvocationHandler and causes
ClassCastException.
h2. Buggy code:
In TestRPC.java:
{code:java}
@Test
public void testWrappedStopProxy() throws IOException {
StoppedProtocol wrappedProxy = RPC.getProxy(StoppedProtocol.class,
StoppedProtocol.versionID, null, conf);
StoppedInvocationHandler invocationHandler = (StoppedInvocationHandler) //
<--- Here causes ClassCastException
Proxy.getInvocationHandler(wrappedProxy);
...
} {code}
h2. How to reproduce:
(1) Just directly run test TestRPC#testWrappedStopProxy.
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
> ClassCastException in test TestRPC#testWrappedStopProxy
> ---
>
> Key: HADOOP-18803
> URL: https://issues.apache.org/jira/browse/HADOOP-18803
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ipc
>Reporter: ConfX
>Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened:
> In HCommon, test TestRPC#testWrappedStopProxy tries to cast
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker to
> org.apache.hadoop.ipc.TestRPC$StoppedInvocationHandler and causes
> ClassCastException.
> h2. Buggy code:
> In TestRPC.java:
> {code:java}
> @Test
> public void testWrappedStopProxy() throws IOException {
> StoppedProtocol wrappedProxy = RPC.getProxy(StoppedProtocol.class,
> StoppedProtocol.versionID, null, conf);
> StoppedInvocationHandler invocationHandler = (StoppedInvocationHandler)
> // <--- Here causes ClassCastException
> Proxy.getInvocationHandler(wrappedProxy);
> ...
> } {code}
> h2. How to reproduce:
> (1) Just directly run test TestRPC#testWrappedStopProxy.
> You can use the reproduce.sh in the attachment to easily reproduce the bug.
> We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18803) ClassCastException in test TestRPC#testWrappedStopProxy
ConfX created HADOOP-18803:
--
Summary: ClassCastException in test TestRPC#testWrappedStopProxy
Key: HADOOP-18803
URL: https://issues.apache.org/jira/browse/HADOOP-18803
Project: Hadoop Common
Issue Type: Bug
Components: common, ipc
Reporter: ConfX
Attachments: reproduce.sh
h2. What happened:
In HCommon, test TestRPC#testWrappedStopProxy tries to cast
org.apache.hadoop.ipc.WritableRpcEngine$Invoker to
org.apache.hadoop.ipc.TestRPC$StoppedInvocationHandler and causes
ClassCastException.
h2. Buggy code:
In TestRPC.java:
{code:java}
@Test
public void testWrappedStopProxy() throws IOException {
StoppedProtocol wrappedProxy = RPC.getProxy(StoppedProtocol.class,
StoppedProtocol.versionID, null, conf);
StoppedInvocationHandler invocationHandler = (StoppedInvocationHandler) //
<--- Here causes ClassCastException
Proxy.getInvocationHandler(wrappedProxy);
...
} {code}
h2. How to reproduce:
(1) Just directly run test TestRPC#testWrappedStopProxy.
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18776) Add OptimizedS3AMagicCommitter For Zero Rename Commits to S3 Endpoints
[
https://issues.apache.org/jira/browse/HADOOP-18776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743161#comment-17743161
]
Syed Shameerur Rahman commented on HADOOP-18776:
[~ste...@apache.org] - If i understood your comment, You are proposing
something like even if this committer(which does complete mpu in commitTask) is
enabled when task attempt retry is 1 then we are okay, If not there should be
some mechanism to fail the job when we use this committer and task attempt
retry > 1 and the task which failed had called commitTask operation
Am i correct?
> Add OptimizedS3AMagicCommitter For Zero Rename Commits to S3 Endpoints
> --
>
> Key: HADOOP-18776
> URL: https://issues.apache.org/jira/browse/HADOOP-18776
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Reporter: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> The goal is to add a new S3A committer named *OptimizedS3AMagicCommitter*
> which is an another type of S3 Magic committer but with a better performance
> by taking in few tradeoffs.
> The following are the differences in MagicCommitter vs OptimizedMagicCommitter
>
> ||Operation||Magic Committer||*OptimizedS3AMagicCommitter*||
> |commitTask|1. Lists all {{.pending}} files in its attempt directory.
>
> 2. The contents are loaded into a list of single pending uploads.
>
> 3. Saved to a {{.pendingset}} file in the job attempt directory.|1. Lists all
> {{.pending}} files in its attempt directory
>
> 2. The contents are loaded into a list of single pending uploads.
>
> 3. For each pending upload, commit operation is called (complete
> multiPartUpload)|
> |commitJob|1. Loads all {{.pendingset}} files in its job attempt directory
>
> 2. Then every pending commit in the job will be committed.
>
> 3. "SUCCESS" marker is created (if config is enabled)
>
> 4. "__magic" directory is cleaned up.|1. "SUCCESS" marker is created (if
> config is enabled)
>
> 2. "__magic" directory is cleaned up.|
>
> *Performance Benefits :-*
> # The primary performance boost due to distributed complete multiPartUpload
> call being made in the taskAttempts(Task containers/Executors) rather than a
> single job driver. In case of MagicCommitter it is O(files/threads).
> # It also saves a couple of S3 calls needed to PUT the "{{{}.pendingset{}}}"
> files and READ call to read them in the Job Driver.
>
> *TradeOffs :-*
> The tradeoffs are similar to the one in FileOutputCommitter V2 version. Users
> migrating from FileOutputCommitter V2 to OptimizedS3AMagicCommitter will no
> see behavioral change as such
> # During execution, intermediate data becomes visible after commitTask
> operation
> # On a failure, all output must be deleted and the job needs to be restarted.
>
> *Performance Benchmark :-*
> Cluster : c4.8x large (ec2-instance)
> Instance : 1 (primary) + 5 (core)
> Data Size : 3TB Partitioned(TPC-DS store_sales data)
> Engine : Apache Spark 3.3.1 / Hadoop 3.3.3
> Query: The following query inserts around 3000+ files into the table
> directory (ran for 3 iterations)
> {code:java}
> insert into select ss_quantity from store_sales; {code}
> ||Committer||Iteration 1||Iteration 2||Iteration 3||
> |Magic|126|127|122|
> |OptimizedMagic|50|51|58|
> So on an average, OptimizedMagicCommitter was *~2.3x* faster as compared to
> MagicCommitter.
>
> _*Note: Unlike MagicCommitter , OptimizedMagicCommitter is not suitable for
> all the cases where in user requires the guarantees of file not being visible
> in failure scenarios. Given the performance benefit, user can may choose to
> use this if they don't require any guarantees or have some mechanism to clean
> up the data before retrying.*_
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18802) Collision of config key name fs.viewfs.mounttable.default.name.key to other keys that specify the entry point to mount tables
[
https://issues.apache.org/jira/browse/HADOOP-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18802:
---
Description:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of InodeTree, the tree attempts to get all the mount table
entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here mountTablePrefix="fs.viewfs.mounttable.default.". However, it just so
happens that the name of the configuration users use to specify the default
mount table is fs.viewfs.mounttable.default.name.key. Thus, if a user specifies
the default mount table and uses InodeTree the name.key would be falsely parsed
as the entry point to one of the mount tables, which would cause InodeTree to
throw an exception since name.key is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set fs.viewfs.mounttable.default.name.key to default
(2) Run test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
was:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of InodeTree, the tree attempts to get all the mount table
entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
> Collision of config key name fs.viewfs.mounttable.default.name.key to other
> keys that specify the entry point to mount tables
> -
>
> Key: HADOOP-18802
> URL: https://issues.apache.org/jira/browse/HADOOP-18802
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, conf, fs
>Reporter: ConfX
>Priority: Critical
> Attachments: reproduce.sh
>
>
> h2.
[jira] [Updated] (HADOOP-18802) Collision of config key name fs.viewfs.mounttable.default.name.key to other keys that specify the entry point to mount tables
[
https://issues.apache.org/jira/browse/HADOOP-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18802:
---
Description:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of InodeTree, the tree attempts to get all the mount table
entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
was:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of {{{}InodeTree{}}}, the tree attempts to get all the mount
table entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
> Collision of config key name fs.viewfs.mounttable.default.name.key to other
> keys that specify the entry point to mount tables
> -
>
> Key: HADOOP-18802
> URL: https://issues.apache.org/jira/browse/HADOOP-18802
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, conf, fs
>Reporter: ConfX
>Priority: Critical
>
[jira] [Updated] (HADOOP-18802) Collision of config key name fs.viewfs.mounttable.default.name.key to other keys that specify the entry point to mount tables
[
https://issues.apache.org/jira/browse/HADOOP-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18802:
---
Description:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of {{{}InodeTree{}}}, the tree attempts to get all the mount
table entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
was:
h2. What happened:
When manually set {{fs.viewfs.mounttable.default.name.key}} to {{default}} (the
same as default value) in HCommon, test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of {{{}InodeTree{}}}, the tree attempts to get all the mount
table entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
{{}}
> Collision of config key name fs.viewfs.mounttable.default.name.key to other
> keys that specify the entry point to mount tables
> -
>
> Key: HADOOP-18802
> URL: https://issues.apache.org/jira/browse/HADOOP-18802
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, conf, fs
>Reporter: ConfX
>
[jira] [Updated] (HADOOP-18802) Collision of config key name fs.viewfs.mounttable.default.name.key to other keys that specify the entry point to mount tables
[
https://issues.apache.org/jira/browse/HADOOP-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18802:
---
Description:
h2. What happened:
When manually set {{fs.viewfs.mounttable.default.name.key}} to {{default}} (the
same as default value) in HCommon, test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of {{{}InodeTree{}}}, the tree attempts to get all the mount
table entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
{{}}
was:
h2. What happened:
When manually set {{fs.viewfs.mounttable.default.name.key}} to {{default}} (the
same as default value) in HCommon, test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of {{{}InodeTree{}}}, the tree attempts to get all the mount
table entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey(); if
(!key.startsWith(mountTablePrefix)) { continue; }
gotMountTableEntry = true; LinkType linkType; String src =
key.substring(mountTablePrefix.length()); ...
{code}
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.keyat
org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470) at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38)
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
{{}}
> Collision of config key name fs.viewfs.mounttable.default.name.key to other
> keys that specify the entry point to mount tables
> -
>
> Key: HADOOP-18802
> URL: https://issues.apache.org/jira/browse/HADOOP-18802
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, conf, fs
>Reporter: ConfX
>Priority: Critical
> Attachments: reproduce.sh
>
[GitHub] [hadoop] hadoop-yetus commented on pull request #5841: YARN-11531. [Federation] Code cleanup for NodeManager#amrmproxy.
hadoop-yetus commented on PR #5841:
URL: https://github.com/apache/hadoop/pull/5841#issuecomment-1635845485
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 56m 50s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5841/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 16 new + 29 unchanged - 1 fixed = 45 total (was 30) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 0s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 176m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5841/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5841 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e3a2ea871790 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f9b09247f5d0e417bcafd2f492247a5d4b3a12a9 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5841/1/testReport/ |
| Max. process+thread count | 539 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
[jira] [Updated] (HADOOP-18802) Collision of config key name fs.viewfs.mounttable.default.name.key to other keys that specify the entry point to mount tables
[
https://issues.apache.org/jira/browse/HADOOP-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18802:
---
Description:
h2. What happened:
When manually set {{fs.viewfs.mounttable.default.name.key}} to {{default}} (the
same as default value) in HCommon, test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of {{{}InodeTree{}}}, the tree attempts to get all the mount
table entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry si : config) {
final String key = si.getKey(); if
(!key.startsWith(mountTablePrefix)) { continue; }
gotMountTableEntry = true; LinkType linkType; String src =
key.substring(mountTablePrefix.length()); ...
{code}
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.keyat
org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470) at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38)
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
{{}}
was:
h2. What happened:
When manually set {{fs.viewfs.mounttable.default.name.key}} to {{default}} (the
same as default value) in HCommon, test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of {{{}InodeTree{}}}, the tree attempts to get all the mount
table entry points set by user in the configuration and process them one by one:
for (Entry si : config) { final String key =
si.getKey(); if (!key.startsWith(mountTablePrefix)) {continue;
} gotMountTableEntry = true; LinkType linkType; String src
= key.substring(mountTablePrefix.length());
...
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.keyat
org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38)
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
{{}}
> Collision of config key name fs.viewfs.mounttable.default.name.key to other
> keys that specify the entry point to mount tables
> -
>
> Key: HADOOP-18802
> URL: https://issues.apache.org/jira/browse/HADOOP-18802
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, conf, fs
>Reporter: ConfX
>Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened:
> When manually set {{fs.viewfs.mounttable.default.name.key}} to
[jira] [Created] (HADOOP-18802) Collision of config key name fs.viewfs.mounttable.default.name.key to other keys that specify the entry point to mount tables
ConfX created HADOOP-18802:
--
Summary: Collision of config key name
fs.viewfs.mounttable.default.name.key to other keys that specify the entry
point to mount tables
Key: HADOOP-18802
URL: https://issues.apache.org/jira/browse/HADOOP-18802
Project: Hadoop Common
Issue Type: Bug
Components: common, conf, fs
Reporter: ConfX
Attachments: reproduce.sh
h2. What happened:
When manually set {{fs.viewfs.mounttable.default.name.key}} to {{default}} (the
same as default value) in HCommon, test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of {{{}InodeTree{}}}, the tree attempts to get all the mount
table entry points set by user in the configuration and process them one by one:
for (Entry si : config) { final String key =
si.getKey(); if (!key.startsWith(mountTablePrefix)) {continue;
} gotMountTableEntry = true; LinkType linkType; String src
= key.substring(mountTablePrefix.length());
...
Here {{{}mountTablePrefix="fs.viewfs.mounttable.default."{}}}. However, it just
so happens that the name of the configuration users use to specify the default
mount table is {{{}fs.viewfs.mounttable.default.name.key{}}}. Thus, if a user
specifies the default mount table and uses InodeTree the {{name.key}} would be
falsely parsed as the entry point to one of the mount tables, which would cause
{{InodeTree}} to throw an exception since {{name.key}} is not a valid entry.
h2. Stack trace:
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.keyat
org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38)
h2. How to reproduce:
(1) Set {{fs.viewfs.mounttable.default.name.key}} to {{default}}
(2) Run test
{{org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches}}
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5840: HDFS-16911. Distcp with snapshot diff to support Ozone filesystem.
hadoop-yetus commented on PR #5840:
URL: https://github.com/apache/hadoop/pull/5840#issuecomment-1635821574
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 68m 20s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 37s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 0m 59s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 36m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 17s |
[/results-checkstyle-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5840/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-distcp.txt)
| hadoop-tools/hadoop-distcp: The patch generated 3 new + 26 unchanged - 0
fixed = 29 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 15s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 176m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5840/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5840 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 53f1e691b8b3 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 9c4236fef821ba694a40efcd30e0073207c592ff |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5840/1/testReport/ |
| Max. process+thread count | 743 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5840/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5839: HDFS-13916. Distcp SnapshotDiff to support WebHDFS.
hadoop-yetus commented on PR #5839:
URL: https://github.com/apache/hadoop/pull/5839#issuecomment-1635812415
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 53m 32s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 26s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 29s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 0m 54s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 40m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed |
| +1 :green_heart: | spotbugs | 0m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 24s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 170m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5839/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5839 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2bd1471917a2 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 87d07dcd1a6b99c995aca32fb8ea8c8db7bda181 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5839/1/testReport/ |
| Max. process+thread count | 592 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5839/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 opened a new pull request, #5842: HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota
haiyang1987 opened a new pull request, #5842: URL: https://github.com/apache/hadoop/pull/5842 ### Description of PR https://issues.apache.org/jira/browse/HDFS-17086 Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] harris233 commented on pull request #5807: HDFS-17080. fix ec connection leak
harris233 commented on PR #5807: URL: https://github.com/apache/hadoop/pull/5807#issuecomment-1635716182 > Approved. Please following the guide of email from ASF. Thanks. Thanks, I have successfully applied -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on a diff in pull request #5823: HDFS-17075. Reconfig disk balancer parameters for datanode
haiyang1987 commented on code in PR #5823:
URL: https://github.com/apache/hadoop/pull/5823#discussion_r1263602582
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -951,6 +960,45 @@ private String reconfDfsUsageParameters(String property,
String newVal)
}
}
+ private String reconfDiskBalancerParameters(String property, String newVal)
+ throws ReconfigurationException {
+String result = null;
+try {
+ LOG.info("Reconfiguring {} to {}", property, newVal);
+ if (property.equals(DFS_DISK_BALANCER_ENABLED)) {
+if (newVal != null && !newVal.equalsIgnoreCase("true")
+&& !newVal.equalsIgnoreCase("false")) {
+ throw new IllegalArgumentException("Not a valid Boolean value for "
+ property +
+ " in reconfDiskBalancerParameters");
+}
+boolean enable = (newVal == null ? DFS_DISK_BALANCER_ENABLED_DEFAULT :
+Boolean.parseBoolean(newVal));
Review Comment:
Hi Sir, the addition of this check is intended to consider the case where
newVal is null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on pull request #5834: HDFS-17082. Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md"
haiyang1987 commented on PR #5834: URL: https://github.com/apache/hadoop/pull/5834#issuecomment-1635648022 Thanks @ayushtkn @slfan1989 help review and merge it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on pull request #5833: HDFS-17081. Append ec file check if a block is replicated to at least the minimum replication need consider striped block
haiyang1987 commented on PR #5833: URL: https://github.com/apache/hadoop/pull/5833#issuecomment-1635646287 Thanks @ayushtkn help review and merge it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on pull request #5823: HDFS-17075. Reconfig disk balancer parameters for datanode
haiyang1987 commented on PR #5823: URL: https://github.com/apache/hadoop/pull/5823#issuecomment-1635644518 Thanks @ayushtkn help me review, i will update it later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 opened a new pull request, #5841: YARN-11531. [Federation] Code cleanup for NodeManager#amrmproxy.
slfan1989 opened a new pull request, #5841: URL: https://github.com/apache/hadoop/pull/5841 ### Description of PR JIRA: YARN-11531. [Federation] Code cleanup for NodeManager#amrmproxy. ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sadanand48 opened a new pull request, #5840: HDFS-16911. Distcp with snapshot diff to support Ozone filesystem.
sadanand48 opened a new pull request, #5840: URL: https://github.com/apache/hadoop/pull/5840 ### Description of PR Backport of HDFS-16911 into branch-3.3 ### How was this patch tested? Unit tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sadanand48 opened a new pull request, #5839: HDFS-13916. Distcp SnapshotDiff to support WebHDFS.
sadanand48 opened a new pull request, #5839: URL: https://github.com/apache/hadoop/pull/5839 ### Description of PR Backport of change HDFS-13916 into branch-3.3 ### How was this patch tested? Unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[ https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743069#comment-17743069 ] ASF GitHub Bot commented on HADOOP-18291: - steveloughran merged PR #5754: URL: https://github.com/apache/hadoop/pull/5754 > S3A prefetch - Implement LRU cache for SingleFilePerBlockCache > -- > > Key: HADOOP-18291 > URL: https://issues.apache.org/jira/browse/HADOOP-18291 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Currently there is no limit on the size of disk cache. This means we could > have a large number of files on files, especially for access patterns that > are very random and do not always read the block fully. > > eg: > in.seek(5); > in.read(); > in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read > in.read(); > in.seek(2 * blockSize + 10) // block 1 gets saved to disk > .. and so on > > The in memory cache is bounded, and by default has a limit of 72MB (9 > blocks). When a block is fully read, and a seek is issued it's released > [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109]. > We can also delete the on disk file for the block here if it exists. > > Also maybe add an upper limit on disk space, and delete the file which stores > data of the block furthest from the current block (similar to the in memory > cache) when this limit is reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
steveloughran merged PR #5754: URL: https://github.com/apache/hadoop/pull/5754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a diff in pull request #5823: HDFS-17075. Reconfig disk balancer parameters for datanode
ayushtkn commented on code in PR #5823:
URL: https://github.com/apache/hadoop/pull/5823#discussion_r1263492781
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -951,6 +960,45 @@ private String reconfDfsUsageParameters(String property,
String newVal)
}
}
+ private String reconfDiskBalancerParameters(String property, String newVal)
+ throws ReconfigurationException {
+String result = null;
+try {
+ LOG.info("Reconfiguring {} to {}", property, newVal);
+ if (property.equals(DFS_DISK_BALANCER_ENABLED)) {
+if (newVal != null && !newVal.equalsIgnoreCase("true")
+&& !newVal.equalsIgnoreCase("false")) {
+ throw new IllegalArgumentException("Not a valid Boolean value for "
+ property +
+ " in reconfDiskBalancerParameters");
Review Comment:
this isn't required
```
" in reconfDiskBalancerParameters"
```
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -4201,7 +4249,7 @@ public List getVolumeReport() throws
IOException {
return volumeInfoList;
}
- private DiskBalancer getDiskBalancer() throws IOException {
+ public DiskBalancer getDiskBalancer() throws IOException {
Review Comment:
Add ```@VisibleForTesting```
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -951,6 +960,45 @@ private String reconfDfsUsageParameters(String property,
String newVal)
}
}
+ private String reconfDiskBalancerParameters(String property, String newVal)
+ throws ReconfigurationException {
+String result = null;
+try {
+ LOG.info("Reconfiguring {} to {}", property, newVal);
+ if (property.equals(DFS_DISK_BALANCER_ENABLED)) {
+if (newVal != null && !newVal.equalsIgnoreCase("true")
+&& !newVal.equalsIgnoreCase("false")) {
+ throw new IllegalArgumentException("Not a valid Boolean value for "
+ property +
+ " in reconfDiskBalancerParameters");
+}
+boolean enable = (newVal == null ? DFS_DISK_BALANCER_ENABLED_DEFAULT :
+Boolean.parseBoolean(newVal));
Review Comment:
this is a redundant check, earlier above we already did checks to figure out
whether it is true/false, should have done check once and saved the value above
only
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java:
##
@@ -341,6 +341,59 @@ private void checkDiskBalancerEnabled()
}
}
+ /**
+ * Sets Disk balancer is to enable or not to enable.
+ *
+ * @param diskBalancerEnabled
+ * true, enable diskBalancer, otherwise false to disable it.
+ */
+ public void setDiskBalancerEnabled(boolean diskBalancerEnabled) {
+isDiskBalancerEnabled = diskBalancerEnabled;
+ }
+
+ /**
+ * Returns the value indicating if diskBalancer is enabled.
+ *
+ * @return boolean.
+ */
+ @VisibleForTesting
+ public boolean isDiskBalancerEnabled() {
+return isDiskBalancerEnabled;
+ }
+
+ /**
+ * Sets maximum amount of time disk balancer plan is valid.
+ *
+ * @param planValidityInterval
Review Comment:
add description for the param
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java:
##
@@ -341,6 +341,59 @@ private void checkDiskBalancerEnabled()
}
}
+ /**
+ * Sets Disk balancer is to enable or not to enable.
+ *
+ * @param diskBalancerEnabled
+ * true, enable diskBalancer, otherwise false to disable it.
+ */
+ public void setDiskBalancerEnabled(boolean diskBalancerEnabled) {
+isDiskBalancerEnabled = diskBalancerEnabled;
+ }
+
+ /**
+ * Returns the value indicating if diskBalancer is enabled.
+ *
+ * @return boolean.
+ */
+ @VisibleForTesting
+ public boolean isDiskBalancerEnabled() {
+return isDiskBalancerEnabled;
+ }
+
+ /**
+ * Sets maximum amount of time disk balancer plan is valid.
+ *
+ * @param planValidityInterval
+ */
+ public void setPlanValidityInterval(long planValidityInterval) {
+
this.config.setTimeDuration(DFSConfigKeys.DFS_DISK_BALANCER_PLAN_VALID_INTERVAL,
+planValidityInterval, TimeUnit.MILLISECONDS);
+this.planValidityInterval = planValidityInterval;
+ }
+
+ /**
+ * Gets maximum amount of time disk balancer plan is valid, then
milliseconds is assumed.
+ *
+ * @return long
Review Comment:
it returns ``plan validity interval``, rather than putting the return type
add description
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java:
##
@@ -341,6 +341,59 @@ private void checkDiskBalancerEnabled()
}
}
+ /**
+ * Sets Disk balancer is to enable or not to enable.
+ *
+ *
[GitHub] [hadoop] hadoop-yetus commented on pull request #5838: fix MAPREDUCE-7442. Fix exception message is not intusive when accessing the job configuration web UI
hadoop-yetus commented on PR #5838:
URL: https://github.com/apache/hadoop/pull/5838#issuecomment-1635539975
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 8m 28s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 150m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5838/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5838 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 17fae0e30cf2 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9c17c57020c927ce2c5d41b246ad68d57a2c3de7 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5838/1/testReport/ |
| Max. process+thread count | 613 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5838/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated
[GitHub] [hadoop] symious commented on pull request #5835: HDFS-17084. Utilize StringTable for numerable XAttributes
symious commented on PR #5835: URL: https://github.com/apache/hadoop/pull/5835#issuecomment-1635539770 @jojochuang @ferhui Could you help to review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5835: HDFS-17084. Utilize StringTable for numerable XAttributes
hadoop-yetus commented on PR #5835:
URL: https://github.com/apache/hadoop/pull/5835#issuecomment-1635537811
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 21s | | trunk passed |
| +1 :green_heart: | compile | 18m 32s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 3s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 23s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 8m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 59s | | the patch passed |
| +1 :green_heart: | compile | 17m 54s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 17m 54s | | the patch passed |
| +1 :green_heart: | javac | 17m 54s | | the patch passed |
| +1 :green_heart: | compile | 17m 0s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 17m 0s | | the patch passed |
| +1 :green_heart: | javac | 17m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 40s | | root: The patch generated
0 new + 677 unchanged - 10 fixed = 677 total (was 687) |
| +1 :green_heart: | mvnsite | 4m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 17s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 23s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 9m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 42m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 1s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 38s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 252m 51s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 541m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5835/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5835 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat |
| uname | Linux e3e550c0a3b8 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / af54087e707af206ab55962eceeb0faf0cc1cb34 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5835/3/testReport/ |
[GitHub] [hadoop] hfutatzhanghb commented on pull request #5597: HDFS-16993. Datanode supports configure TopN DatanodeNetworkCounts
hfutatzhanghb commented on PR #5597: URL: https://github.com/apache/hadoop/pull/5597#issuecomment-1635378755 > @hfutatzhanghb try writing a minimal test, even if it fails due to same ip, I can try help fixing it from there @ayushtkn Thanx sir. Will write an unit test soonly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #5597: HDFS-16993. Datanode supports configure TopN DatanodeNetworkCounts
ayushtkn commented on PR #5597: URL: https://github.com/apache/hadoop/pull/5597#issuecomment-1635373215 @hfutatzhanghb try writing a minimal test, even if it fails due to same ip, I can try help fixing it from there -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #5809: HDFS-17068. Datanode should record last directory scan time.
ayushtkn commented on PR #5809: URL: https://github.com/apache/hadoop/pull/5809#issuecomment-1635349696 @Hexiaoqiao any further comments, planning to hold for you a couple of days -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liangyu-1 opened a new pull request, #5838: fix MAPREDUCE-7442. Fix exception message is not intusive when accessing the job configuration web UI
liangyu-1 opened a new pull request, #5838: URL: https://github.com/apache/hadoop/pull/5838 ### Description of PR fix the bug is issue [MAPREDUCE-7442](https://issues.apache.org/jira/browse/MAPREDUCE-7442) this PR fix the bug that exception message is not intusive when accessing the job configuration web UI. ### How was this patch tested? I rebuild the project, and restart our own Hadoop cluster, then we can the exception message on the webpage of the job configuration, and here is the picture:  ### For code changes: I changed the file hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java, line116. - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #5833: HDFS-17081. Append ec file check if a block is replicated to at least the minimum replication need consider striped block
ayushtkn merged PR #5833: URL: https://github.com/apache/hadoop/pull/5833 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #5836: HDFS-17083. Support getErasureCodeCodecs API in WebHDFS
ayushtkn merged PR #5836: URL: https://github.com/apache/hadoop/pull/5836 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
