[jira] [Commented] (HADOOP-15857) Remove ozonefs class name definition from core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651482#comment-16651482 ] Shashikant Banerjee commented on HADOOP-15857: -- +1 > Remove ozonefs class name definition from core-default.xml > -- > > Key: HADOOP-15857 > URL: https://issues.apache.org/jira/browse/HADOOP-15857 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > Attachments: HADOOP-15857-branch-3.2.001.patch > > > Ozone file system is under renaming in HDDS-651 from o3:// to o3fs://. But > branch-3.2 still contains a reference with o3://. > The easiest way to fix it just remove the fs.o3.imp definition from > core-default.xml from branch-3.2 as since HDDS-654 the file system could be > registered with Service Provider Interface (META-INF/services...) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15637) LocalFs#listLocatedStatus does not filter out hidden .crc files
[
https://issues.apache.org/jira/browse/HADOOP-15637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561578#comment-16561578
]
Shashikant Banerjee edited comment on HADOOP-15637 at 7/30/18 7:48 AM:
---
Thanks [~xkrogen], for reporting and workin on this. I do agree with
[~xkrogen], on this. +1 on the patch v0.
was (Author: shashikant):
Thanks [~xkrogen], for reporting and workin on this. I do agree with
[~xkrogen], on this. +1.
> LocalFs#listLocatedStatus does not filter out hidden .crc files
> ---
>
> Key: HADOOP-15637
> URL: https://issues.apache.org/jira/browse/HADOOP-15637
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-15637.000.patch
>
>
> After HADOOP-7165, {{LocalFs#listLocatedStatus}} incorrectly returns the
> hidden {{.crc}} files used to store checksum information. This is because
> HADOOP-7165 implemented {{listLocatedStatus}} on {{FilterFs}}, so the default
> implementation is no longer used, and {{FilterFs}} directly calls the raw FS
> since {{listLocatedStatus}} is not defined in {{ChecksumFs}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15637) LocalFs#listLocatedStatus does not filter out hidden .crc files
[
https://issues.apache.org/jira/browse/HADOOP-15637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561578#comment-16561578
]
Shashikant Banerjee commented on HADOOP-15637:
--
Thanks [~xkrogen], for reporting and workin on this. I do agree with
[~xkrogen], on this. +1.
> LocalFs#listLocatedStatus does not filter out hidden .crc files
> ---
>
> Key: HADOOP-15637
> URL: https://issues.apache.org/jira/browse/HADOOP-15637
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-15637.000.patch
>
>
> After HADOOP-7165, {{LocalFs#listLocatedStatus}} incorrectly returns the
> hidden {{.crc}} files used to store checksum information. This is because
> HADOOP-7165 implemented {{listLocatedStatus}} on {{FilterFs}}, so the default
> implementation is no longer used, and {{FilterFs}} directly calls the raw FS
> since {{listLocatedStatus}} is not defined in {{ChecksumFs}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15548) Randomize local dirs
[ https://issues.apache.org/jira/browse/HADOOP-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527359#comment-16527359 ] Shashikant Banerjee commented on HADOOP-15548: -- +1, LGTM. > Randomize local dirs > > > Key: HADOOP-15548 > URL: https://issues.apache.org/jira/browse/HADOOP-15548 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: HADOOP-15548.001.patch, HADOOP-15548.002.patch > > > shuffle LOCAL_DIRS, LOG_DIRS and LOCAL_USER_DIRS when launching container. > Some applications will process these in exactly the same way in every > container (e.g. roundrobin) which can cause disks to get unnecessarily > overloaded (e.g. one output file written to first entry specified in the > environment variable). > There are two paths for local dir allocation, depending on whether the size > is unknown or known. The unknown path already uses a random algorithm. The > known path initializes with a random starting point, and then goes > round-robin after that. When selecting a dir, it increments the last used by > one and then checks sequentially until it finds a dir that satisfies the > request. Proposal is to increment by a random value of between 1 and > num_dirs - 1, and then check sequentially from there. This should result in > a more random selection in all cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442619#comment-16442619
]
Shashikant Banerjee commented on HADOOP-15396:
--
patch vo fixes the permission for the java files with wrong permissions.
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15396.000.patch
>
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HADOOP-15396:
-
Attachment: HADOOP-15396.000.patch
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15396.000.patch
>
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HADOOP-15396:
-
Status: Patch Available (was: Open)
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15396.000.patch
>
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HADOOP-15396:
Assignee: Shashikant Banerjee
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15366) Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
[
https://issues.apache.org/jira/browse/HADOOP-15366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HADOOP-15366:
-
Status: Patch Available (was: Open)
> Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
>
>
> Key: HADOOP-15366
> URL: https://issues.apache.org/jira/browse/HADOOP-15366
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Minor
> Attachments: HADOOP-15366.000.patch
>
>
> It is recommended to shut down an {{ExecutorService}} in two phases, first by
> calling {{shutdown}} to reject incoming tasks, and then calling
> {{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims
> to add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15366) Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
[
https://issues.apache.org/jira/browse/HADOOP-15366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427508#comment-16427508
]
Shashikant Banerjee commented on HADOOP-15366:
--
Patch v0 adds a helper shutdown routine for hadoop executor service.
> Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
>
>
> Key: HADOOP-15366
> URL: https://issues.apache.org/jira/browse/HADOOP-15366
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Minor
> Attachments: HADOOP-15366.000.patch
>
>
> It is recommended to shut down an {{ExecutorService}} in two phases, first by
> calling {{shutdown}} to reject incoming tasks, and then calling
> {{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims
> to add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15366) Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
[
https://issues.apache.org/jira/browse/HADOOP-15366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HADOOP-15366:
-
Attachment: HADOOP-15366.000.patch
> Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
>
>
> Key: HADOOP-15366
> URL: https://issues.apache.org/jira/browse/HADOOP-15366
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Minor
> Attachments: HADOOP-15366.000.patch
>
>
> It is recommended to shut down an {{ExecutorService}} in two phases, first by
> calling {{shutdown}} to reject incoming tasks, and then calling
> {{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims
> to add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15366) Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
Shashikant Banerjee created HADOOP-15366:
Summary: Add a helper shutDown routine in HadoopExecutor to ensure
clean shutdown
Key: HADOOP-15366
URL: https://issues.apache.org/jira/browse/HADOOP-15366
Project: Hadoop Common
Issue Type: Bug
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
It is recommended to shut down an {{ExecutorService}} in two phases, first by
calling {{shutdown}} to reject incoming tasks, and then calling
{{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims to
add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15074) SequenceFile#Writer flush does not update the length of the written file.
[
https://issues.apache.org/jira/browse/HADOOP-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406096#comment-16406096
]
Shashikant Banerjee commented on HADOOP-15074:
--
I propose the following change in the DFSOutputStream#hsync api to ensure the
file size gets updated at the namenode when hsync completes.
{code:java}
@Override
public void hsync() throws IOException {
try (TraceScope ignored = dfsClient.newPathTraceScope("hsync", src)) {
flushOrSync(true, EnumSet.of(SyncFlag.UPDATE_LENGTH));
}
}{code}
[~arpitagarwal]/others, please let me know your opinion on this.
> SequenceFile#Writer flush does not update the length of the written file.
> -
>
> Key: HADOOP-15074
> URL: https://issues.apache.org/jira/browse/HADOOP-15074
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
>
> SequenceFile#Writer flush does not update the length of the file. This
> happens because as part of the flush, {{UPDATE_LENGTH}} flag is not passed to
> the DFSOutputStream#hsync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
