[PR] HDFS-16984. Directory timestamp lost during the upgrade process [hadoop]

2023-10-11 Thread via GitHub
hanke580 opened a new pull request, #6175: URL: https://github.com/apache/hadoop/pull/6175 ### Description of PR During the upgrade process, when creating the FSImage the access time of the directory is not serialized to the FSImage. This causes the access timestamp lost.

Re: [PR] YARN-11498 backport jettison exclusions [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6063: URL: https://github.com/apache/hadoop/pull/6063#issuecomment-1758865584 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18885) Add rpcRejectedByObserverCalls metric to quantify the number of rejected RPCs by Observer NameNode

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774306#comment-17774306 ] ASF GitHub Bot commented on HADOOP-18885: - haiyang1987 commented on PR #6043: URL:

Re: [PR] HADOOP-18885. Add rpcCallsRejectedByObserver metric to quantify the number of rejected RPCs by Observer NameNode [hadoop]

2023-10-11 Thread via GitHub
haiyang1987 commented on PR #6043: URL: https://github.com/apache/hadoop/pull/6043#issuecomment-1758844199 Hi @goiri @simbadzina excuse me, could you mind to push this modification forward when you have free time ? Thank you very much. -- This is an automated message from the Apache Git

[PR] optimize AvailableSpaceVolumeChoosingPolicy [hadoop]

2023-10-11 Thread via GitHub
GuoPhilipse opened a new pull request, #6174: URL: https://github.com/apache/hadoop/pull/6174 ### Description of PR if all the volumes have the same available space. for example ```1 MB``` and ```dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold``` is set 0,

[jira] [Commented] (HADOOP-18916) module-info classes from external dependencies appearing in uber jars

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774268#comment-17774268 ] ASF GitHub Bot commented on HADOOP-18916: - hadoop-yetus commented on PR #6131: URL:

Re: [PR] HADOOP-18916. Exclude all module-info classes [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6131: URL: https://github.com/apache/hadoop/pull/6131#issuecomment-1758724486 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] YARN-11588. [Federation] Fix uncleaned threads in yarn router thread pool executor [hadoop]

2023-10-11 Thread via GitHub
jchanggg commented on PR #6159: URL: https://github.com/apache/hadoop/pull/6159#issuecomment-1758693970 Thanks for helping review folks. If there are no other issues, could we merge in the patch? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] HADOOP-18889. S3A v2 SDK third party support [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1758604805 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774247#comment-17774247 ] ASF GitHub Bot commented on HADOOP-18889: - hadoop-yetus commented on PR #6141: URL:

Re: [PR] HDFS-17215. RBF: fix some method annotations about @throws . [hadoop]

2023-10-11 Thread via GitHub
ayushtkn commented on code in PR #6136: URL: https://github.com/apache/hadoop/pull/6136#discussion_r1355794059 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/FederationProtocolPBTranslator.java: ## @@ -54,7

Re: [PR] HDFS-17196. Overflow during getDatanodeReadTimeout [hadoop]

2023-10-11 Thread via GitHub
ayushtkn commented on code in PR #6091: URL: https://github.com/apache/hadoop/pull/6091#discussion_r1355790939 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java: ## @@ -477,7 +477,10 @@ int getDatanodeWriteTimeout(int numNodes) {

[jira] [Updated] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18933: Labels: pull-request-available (was: ) > upgrade netty to 4.1.100 due to CVE >

[PR] HADOOP-18933. upgrade to netty 4.1.100 due to CVE [hadoop]

2023-10-11 Thread via GitHub
pjfanning opened a new pull request, #6173: URL: https://github.com/apache/hadoop/pull/6173 ### Description of PR https://issues.apache.org/jira/browse/HADOOP-18933 ### How was this patch tested? ### For code changes: - [x] Does the title or this PR starts

[jira] [Commented] (HADOOP-18929) Build failure while trying to create apache 3.3.7 release locally.

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774232#comment-17774232 ] ASF GitHub Bot commented on HADOOP-18929: - mukund-thakur closed pull request #6169:

[jira] [Commented] (HADOOP-18929) Build failure while trying to create apache 3.3.7 release locally.

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774231#comment-17774231 ] ASF GitHub Bot commented on HADOOP-18929: - mukund-thakur commented on PR #6169: URL:

Re: [PR] HADOOP-18929. Exclude commons-compress module-info.class [hadoop]

2023-10-11 Thread via GitHub
mukund-thakur closed pull request #6169: HADOOP-18929. Exclude commons-compress module-info.class URL: https://github.com/apache/hadoop/pull/6169 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] HADOOP-18929. Exclude commons-compress module-info.class [hadoop]

2023-10-11 Thread via GitHub
mukund-thakur commented on PR #6169: URL: https://github.com/apache/hadoop/pull/6169#issuecomment-1758490824 I have actually merged this https://github.com/apache/hadoop/commits/branch-3.3 just now. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] YARN-9013. [BackPort] [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner. [hadoop]

2023-10-11 Thread via GitHub
goiri commented on code in PR #6147: URL: https://github.com/apache/hadoop/pull/6147#discussion_r1355745625 ##

[jira] [Commented] (HADOOP-18930) make fs.s3a.create.performance an option you can set for the entire bucket

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774220#comment-17774220 ] ASF GitHub Bot commented on HADOOP-18930: - xinglin commented on code in PR #6168: URL:

Re: [PR] HADOOP-18930. Make fs.s3a.create.performance a bucket-wide settings. [hadoop]

2023-10-11 Thread via GitHub
xinglin commented on code in PR #6168: URL: https://github.com/apache/hadoop/pull/6168#discussion_r1355506430 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java: ## @@ -1183,12 +1183,17 @@ private Constants() { /** * Flag for create

[jira] [Commented] (HADOOP-18930) make fs.s3a.create.performance an option you can set for the entire bucket

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774218#comment-17774218 ] ASF GitHub Bot commented on HADOOP-18930: - xinglin commented on code in PR #6168: URL:

Re: [PR] HADOOP-18930. Make fs.s3a.create.performance a bucket-wide settings. [hadoop]

2023-10-11 Thread via GitHub
xinglin commented on code in PR #6168: URL: https://github.com/apache/hadoop/pull/6168#discussion_r1355506430 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java: ## @@ -1183,12 +1183,17 @@ private Constants() { /** * Flag for create

[jira] [Commented] (HADOOP-18929) Build failure while trying to create apache 3.3.7 release locally.

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774209#comment-17774209 ] ASF GitHub Bot commented on HADOOP-18929: - pjfanning commented on PR #6169: URL:

Re: [PR] HADOOP-18929. Exclude commons-compress module-info.class [hadoop]

2023-10-11 Thread via GitHub
pjfanning commented on PR #6169: URL: https://github.com/apache/hadoop/pull/6169#issuecomment-1758404913 @mukund-thakur the 'patch' build seems to have worked ok - do you think this is safe to merge to branch-3.3? -- This is an automated message from the Apache Git Service. To respond to

[jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE

2023-10-11 Thread PJ Fanning (Jira)
PJ Fanning created HADOOP-18933: --- Summary: upgrade netty to 4.1.100 due to CVE Key: HADOOP-18933 URL: https://issues.apache.org/jira/browse/HADOOP-18933 Project: Hadoop Common Issue Type:

[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774196#comment-17774196 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on PR #6141: URL:

Re: [PR] HADOOP-18889. S3A v2 SDK third party support [hadoop]

2023-10-11 Thread via GitHub
steveloughran commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1758298518 once javadocs is happy i will merge this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Commented] (HADOOP-18930) make fs.s3a.create.performance an option you can set for the entire bucket

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774174#comment-17774174 ] ASF GitHub Bot commented on HADOOP-18930: - xinglin commented on code in PR #6168: URL:

Re: [PR] HADOOP-18930. Make fs.s3a.create.performance a bucket-wide settings. [hadoop]

2023-10-11 Thread via GitHub
xinglin commented on code in PR #6168: URL: https://github.com/apache/hadoop/pull/6168#discussion_r1355506430 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java: ## @@ -1183,12 +1183,17 @@ private Constants() { /** * Flag for create

[jira] [Assigned] (HADOOP-18927) S3ARetryHandler to treat SocketExceptions as connectivity failures

2023-10-11 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18927: --- Assignee: Steve Loughran > S3ARetryHandler to treat SocketExceptions as

[jira] [Created] (HADOOP-18932) upgrade AWS v1 and v2 SDKs

2023-10-11 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-18932: --- Summary: upgrade AWS v1 and v2 SDKs Key: HADOOP-18932 URL: https://issues.apache.org/jira/browse/HADOOP-18932 Project: Hadoop Common Issue Type:

[jira] [Commented] (HADOOP-18929) Build failure while trying to create apache 3.3.7 release locally.

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774164#comment-17774164 ] ASF GitHub Bot commented on HADOOP-18929: - mukund-thakur merged PR #6170: URL:

Re: [PR] HADOOP-18929. Exclude commons-compress module-info.class [hadoop]

2023-10-11 Thread via GitHub
mukund-thakur merged PR #6170: URL: https://github.com/apache/hadoop/pull/6170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HADOOP-18908) Improve s3a region handling, including determining from endpoint

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774157#comment-17774157 ] ASF GitHub Bot commented on HADOOP-18908: - hadoop-yetus commented on PR #6106: URL:

Re: [PR] HADOOP-18908. Improve S3A region handling. [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6106: URL: https://github.com/apache/hadoop/pull/6106#issuecomment-1758174153 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18908) Improve s3a region handling, including determining from endpoint

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774154#comment-17774154 ] ASF GitHub Bot commented on HADOOP-18908: - hadoop-yetus commented on PR #6106: URL:

Re: [PR] HADOOP-18908. Improve S3A region handling. [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6106: URL: https://github.com/apache/hadoop/pull/6106#issuecomment-1758169180 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774146#comment-17774146 ] ASF GitHub Bot commented on HADOOP-18889: - hadoop-yetus commented on PR #6141: URL:

Re: [PR] HADOOP-18889. S3A v2 SDK third party support [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1758142645 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] HADOOP-18908. Improve S3A region handling. [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6106: URL: https://github.com/apache/hadoop/pull/6106#issuecomment-1758126370 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18908) Improve s3a region handling, including determining from endpoint

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774144#comment-17774144 ] ASF GitHub Bot commented on HADOOP-18908: - hadoop-yetus commented on PR #6106: URL:

Re: [PR] HDFS-16740. Mini cluster test flakiness [hadoop]

2023-10-11 Thread via GitHub
snmvaughan commented on PR #4835: URL: https://github.com/apache/hadoop/pull/4835#issuecomment-1758084857 Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[jira] [Created] (HADOOP-18931) FileSystem.getFileSystemClass() to log at debug the jar the .class came from

2023-10-11 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-18931: --- Summary: FileSystem.getFileSystemClass() to log at debug the jar the .class came from Key: HADOOP-18931 URL: https://issues.apache.org/jira/browse/HADOOP-18931

Re: [PR] HADOOP-18908. Improve S3A region handling. [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6106: URL: https://github.com/apache/hadoop/pull/6106#issuecomment-1757934033 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18908) Improve s3a region handling, including determining from endpoint

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774112#comment-17774112 ] ASF GitHub Bot commented on HADOOP-18908: - hadoop-yetus commented on PR #6106: URL:

[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774100#comment-17774100 ] ASF GitHub Bot commented on HADOOP-18708: - steveloughran commented on code in PR #6164: URL:

Re: [PR] HADOOP-18708. Adds in support for client side encryption. [hadoop]

2023-10-11 Thread via GitHub
steveloughran commented on code in PR #6164: URL: https://github.com/apache/hadoop/pull/6164#discussion_r1355135994 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java: ## @@ -82,6 +83,31 @@ S3Client createS3Client(URI uri, S3AsyncClient

[jira] [Commented] (HADOOP-18925) S3A: add option "fs.s3a.copy.from.local.enabled" to enable/disable CopyFromLocalOperation

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774082#comment-17774082 ] ASF GitHub Bot commented on HADOOP-18925: - hadoop-yetus commented on PR #6163: URL:

Re: [PR] HADOOP-18925. S3A: option "fs.s3a.optimized.copy.from.local.enabled" to control CopyFromLocalOperation [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6163: URL: https://github.com/apache/hadoop/pull/6163#issuecomment-1757756363 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] YARN-11590. RM process stuck after calling confStore.format() when ZK… [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6166: URL: https://github.com/apache/hadoop/pull/6166#issuecomment-1757739418 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] YARN-11590. RM process stuck after calling confStore.format() when ZK… [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6166: URL: https://github.com/apache/hadoop/pull/6166#issuecomment-1757669964 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18830) Cut S3 Select

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774054#comment-17774054 ] ASF GitHub Bot commented on HADOOP-18830: - hadoop-yetus commented on PR #6144: URL:

Re: [PR] HADOOP-18830. Cut S3 Select [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6144: URL: https://github.com/apache/hadoop/pull/6144#issuecomment-1757656449 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] YARN-11216. Avoid unnecessary reconstruction of ConfigurationProperties [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #4655: URL: https://github.com/apache/hadoop/pull/4655#issuecomment-1757622191 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] HADOOP-18930. Make fs.s3a.create.performance a bucket-wide settings. [hadoop]

2023-10-11 Thread via GitHub
steveloughran commented on PR #6168: URL: https://github.com/apache/hadoop/pull/6168#issuecomment-1757593346 testing: updated test against s3 london from ide and maven. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[jira] [Commented] (HADOOP-18930) make fs.s3a.create.performance an option you can set for the entire bucket

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774044#comment-17774044 ] ASF GitHub Bot commented on HADOOP-18930: - steveloughran commented on PR #6168: URL:

[jira] [Commented] (HADOOP-18925) S3A: add option "fs.s3a.copy.from.local.enabled" to enable/disable CopyFromLocalOperation

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774043#comment-17774043 ] ASF GitHub Bot commented on HADOOP-18925: - steveloughran commented on PR #6163: URL:

Re: [PR] HADOOP-18925. S3A: option "fs.s3a.optimized.copy.from.local.enabled" to control CopyFromLocalOperation [hadoop]

2023-10-11 Thread via GitHub
steveloughran commented on PR #6163: URL: https://github.com/apache/hadoop/pull/6163#issuecomment-1757589338 @mehakmeet addressed your issues, option is now fs.s3a.optimized.copy.from.local.enabled + improved span of operation + state exported as path capability, which showed the

[jira] [Commented] (HADOOP-18925) S3A: add option "fs.s3a.copy.from.local.enabled" to enable/disable CopyFromLocalOperation

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774042#comment-17774042 ] ASF GitHub Bot commented on HADOOP-18925: - steveloughran commented on code in PR #6163: URL:

Re: [PR] HADOOP-18925. S3A: option "fs.s3a.optimized.copy.from.local.enabled" to control CopyFromLocalOperation [hadoop]

2023-10-11 Thread via GitHub
steveloughran commented on code in PR #6163: URL: https://github.com/apache/hadoop/pull/6163#discussion_r1354892764 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java: ## @@ -4086,7 +4103,8 @@ public void copyLocalFileFromTo(File file, Path

[jira] [Commented] (HADOOP-18930) make fs.s3a.create.performance an option you can set for the entire bucket

2023-10-11 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774038#comment-17774038 ] Steve Loughran commented on HADOOP-18930: - thinking of making the path capability probe show

Re: [PR] HDFS-17210. Optimize AvailableSpaceBlockPlacementPolicy. [hadoop]

2023-10-11 Thread via GitHub
GuoPhilipse commented on code in PR #6113: URL: https://github.com/apache/hadoop/pull/6113#discussion_r1354883983 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java: ## @@ -1240,6 +1240,12 @@ public class DFSConfigKeys extends

Re: [PR] HDFS-17210. Optimize AvailableSpaceBlockPlacementPolicy. [hadoop]

2023-10-11 Thread via GitHub
Hexiaoqiao commented on code in PR #6113: URL: https://github.com/apache/hadoop/pull/6113#discussion_r1354847751 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java: ## @@ -1240,6 +1240,12 @@ public class DFSConfigKeys extends

[jira] [Commented] (HADOOP-18925) S3A: add option "fs.s3a.copy.from.local.enabled" to enable/disable CopyFromLocalOperation

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773970#comment-17773970 ] ASF GitHub Bot commented on HADOOP-18925: - steveloughran commented on code in PR #6163: URL:

Re: [PR] HADOOP-18925. S3A: option "fs.s3a.copy.from.local.enabled" to enable/disable CopyFromLocalOperation [hadoop]

2023-10-11 Thread via GitHub
steveloughran commented on code in PR #6163: URL: https://github.com/apache/hadoop/pull/6163#discussion_r1354492562 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java: ## @@ -1309,4 +1309,17 @@ private Constants() { * The bucket region header.

[jira] [Commented] (HADOOP-18925) S3A: add option "fs.s3a.copy.from.local.enabled" to enable/disable CopyFromLocalOperation

2023-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773969#comment-17773969 ] ASF GitHub Bot commented on HADOOP-18925: - mehakmeet commented on code in PR #6163: URL:

Re: [PR] HADOOP-18925. S3A: option "fs.s3a.copy.from.local.enabled" to enable/disable CopyFromLocalOperation [hadoop]

2023-10-11 Thread via GitHub
mehakmeet commented on code in PR #6163: URL: https://github.com/apache/hadoop/pull/6163#discussion_r1354479653 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java: ## @@ -1309,4 +1309,17 @@ private Constants() { * The bucket region header.

Re: [PR] YARN-11589. Router Policy support user as default queue. [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6171: URL: https://github.com/apache/hadoop/pull/6171#issuecomment-1757173016 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] MAPREDUCE-7458. Race condition in TaskReportPBImpl#getProto when generating task reports process in concurrency scenarios. [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6172: URL: https://github.com/apache/hadoop/pull/6172#issuecomment-1757044374 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] HDFS-17210. Optimize AvailableSpaceBlockPlacementPolicy. [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6113: URL: https://github.com/apache/hadoop/pull/6113#issuecomment-1757018651 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] YARN-11571. [GPG] Add Information About YARN GPG in Federation.md [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6158: URL: https://github.com/apache/hadoop/pull/6158#issuecomment-1757016102 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] MAPREDUCE-7457: Added support to limit count of spill files [hadoop]

2023-10-11 Thread via GitHub
mudit1289 commented on PR #6155: URL: https://github.com/apache/hadoop/pull/6155#issuecomment-1756929923 sure, @slfan1989 thanks for the comment, also I have assumed that code for writing new spills is present in only MapTask class, please let me know if we spill data from some other flow

[PR] MAPREDUCE-7458. Race condition in TaskReportPBImpl#getProto when generating task reports process in concurrency scenarios. [hadoop]

2023-10-11 Thread via GitHub
TaoYang526 opened a new pull request, #6172: URL: https://github.com/apache/hadoop/pull/6172 ### Description of PR Please refer to the description in MAPREDUCE-7458 ### How was this patch tested? No, the exception scenario is hard to reproduce, and the patch is a tiny

Re: [PR] YARN-11589. Router Policy support user as default queue. [hadoop]

2023-10-11 Thread via GitHub
hadoop-yetus commented on PR #6171: URL: https://github.com/apache/hadoop/pull/6171#issuecomment-1756868634 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |