[jira] [Resolved] (YARN-10706) Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2
[ https://issues.apache.org/jira/browse/YARN-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved YARN-10706. -- Fix Version/s: 3.2.3 3.3.1 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to Hadoop 3.x branches. Thank you for your review [~ayushtkn] and [~aajisaka]! > Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 > --- > > Key: YARN-10706 > URL: https://issues.apache.org/jira/browse/YARN-10706 > Project: Hadoop YARN > Issue Type: Bug > Components: buid >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Time Spent: 0.5h > Remaining Estimate: 0h > > It has been years since the com.github.eirslett:frontend-maven-plugin plugin > was brought in. According to its [release > notes|https://github.com/eirslett/frontend-maven-plugin/blob/master/CHANGELOG.md], > recent versions have bug fixes and support more platforms. Specially, it > also supports the Apple Silicon chip so we can build Hadoop on latest macOS > platforms. > This is to upgrade this plugin to the latest published version 1.11.2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10706) Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2
[ https://issues.apache.org/jira/browse/YARN-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305361#comment-17305361 ] Mingliang Liu commented on YARN-10706: -- FWIW, this is the wiki page for building Hadoop on Apple M1. https://cwiki.apache.org/confluence/display/HADOOP/Develop+on+Apple+Silicon+%28M1%29+macOS > Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 > --- > > Key: YARN-10706 > URL: https://issues.apache.org/jira/browse/YARN-10706 > Project: Hadoop YARN > Issue Type: Bug > Components: buid >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > It has been years since the com.github.eirslett:frontend-maven-plugin plugin > was brought in. According to its [release > notes|https://github.com/eirslett/frontend-maven-plugin/blob/master/CHANGELOG.md], > recent versions have bug fixes and support more platforms. Specially, it > also supports the Apple Silicon chip so we can build Hadoop on latest macOS > platforms. > This is to upgrade this plugin to the latest published version 1.11.2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10706) Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2
[ https://issues.apache.org/jira/browse/YARN-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reassigned YARN-10706: Assignee: Mingliang Liu > Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 > --- > > Key: YARN-10706 > URL: https://issues.apache.org/jira/browse/YARN-10706 > Project: Hadoop YARN > Issue Type: Bug > Components: buid >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Major > > It has been years since the com.github.eirslett:frontend-maven-plugin plugin > was brought in. According to its [release > notes|https://github.com/eirslett/frontend-maven-plugin/blob/master/CHANGELOG.md], > recent versions have bug fixes and support more platforms. Specially, it > also supports the Apple Silicon chip so we can build Hadoop on latest macOS > platforms. > This is to upgrade this plugin to the latest published version 1.11.2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10706) Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2
[ https://issues.apache.org/jira/browse/YARN-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated YARN-10706: - Summary: Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 (was: Upgrade com.github.eirslett:frontend-maven-plugin to 1.10) > Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 > --- > > Key: YARN-10706 > URL: https://issues.apache.org/jira/browse/YARN-10706 > Project: Hadoop YARN > Issue Type: Bug > Components: buid >Reporter: Mingliang Liu >Priority: Major > > It has been years since the com.github.eirslett:frontend-maven-plugin plugin > was brought in. According to its [release > notes|https://github.com/eirslett/frontend-maven-plugin/blob/master/CHANGELOG.md], > recent versions have bug fixes and support more platforms. Specially, it > also supports the Apple Silicon chip so we can build Hadoop on latest macOS > platforms. > This is to upgrade this plugin to the latest published version 1.11.2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10706) Upgrade com.github.eirslett:frontend-maven-plugin to 1.10
Mingliang Liu created YARN-10706: Summary: Upgrade com.github.eirslett:frontend-maven-plugin to 1.10 Key: YARN-10706 URL: https://issues.apache.org/jira/browse/YARN-10706 Project: Hadoop YARN Issue Type: Bug Components: buid Reporter: Mingliang Liu It has been years since the com.github.eirslett:frontend-maven-plugin plugin was brought in. According to its [release notes|https://github.com/eirslett/frontend-maven-plugin/blob/master/CHANGELOG.md], recent versions have bug fixes and support more platforms. Specially, it also supports the Apple Silicon chip so we can build Hadoop on latest macOS platforms. This is to upgrade this plugin to the latest published version 1.11.2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9768) RM Renew Delegation token thread should timeout and retry
[ https://issues.apache.org/jira/browse/YARN-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027797#comment-17027797 ] Mingliang Liu commented on YARN-9768: - Thanks [~inigoiri]. Yes, this is good now! > RM Renew Delegation token thread should timeout and retry > - > > Key: YARN-9768 > URL: https://issues.apache.org/jira/browse/YARN-9768 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: CR Hota >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9768.001.patch, YARN-9768.002.patch, > YARN-9768.003.patch, YARN-9768.004.patch, YARN-9768.005.patch, > YARN-9768.006.patch, YARN-9768.007.patch, YARN-9768.008.patch, > YARN-9768.009.patch, YARN-9768.010.patch > > > Delegation token renewer thread in RM (DelegationTokenRenewer.java) renews > HDFS tokens received to check for validity and expiration time. > This call is made to an underlying HDFS NN or Router Node (which has exact > APIs as HDFS NN). If one of the nodes is bad and the renew call is stuck the > thread remains stuck indefinitely. The thread should ideally timeout the > renewToken and retry from the client's perspective. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9768) RM Renew Delegation token thread should timeout and retry
[
https://issues.apache.org/jira/browse/YARN-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020712#comment-17020712
]
Mingliang Liu commented on YARN-9768:
-
Thanks [~elgoiri] for prompt reply and fix. The {{trunk}} branch is good to me
now.
> RM Renew Delegation token thread should timeout and retry
> -
>
> Key: YARN-9768
> URL: https://issues.apache.org/jira/browse/YARN-9768
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: CR Hota
>Assignee: Manikandan R
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9768.001.patch, YARN-9768.002.patch,
> YARN-9768.003.patch, YARN-9768.004.patch, YARN-9768.005.patch,
> YARN-9768.006.patch, YARN-9768.007.patch, YARN-9768.008.patch
>
>
> Delegation token renewer thread in RM (DelegationTokenRenewer.java) renews
> HDFS tokens received to check for validity and expiration time.
> This call is made to an underlying HDFS NN or Router Node (which has exact
> APIs as HDFS NN). If one of the nodes is bad and the renew call is stuck the
> thread remains stuck indefinitely. The thread should ideally timeout the
> renewToken and retry from the client's perspective.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9768) RM Renew Delegation token thread should timeout and retry
[
https://issues.apache.org/jira/browse/YARN-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020698#comment-17020698
]
Mingliang Liu commented on YARN-9768:
-
I saw errors like this on {{trunk}} branch:
{quote}
$ mvn clean package -DskipTests -q
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3/libexec
Java version: 1.8.0_162, vendor: Oracle Corporation, runtime:
/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.2", arch: "x86_64", family: "mac"
[ERROR] COMPILATION ERROR :
[ERROR]
/Users/mingliang.liu/Workspace/apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java:[1675,7]
cannot find symbol
symbol: method
submitApp(int,java.lang.String,java.lang.String,java.util.HashMap,boolean,java.lang.String,int,org.apache.hadoop.security.Credentials)
location: variable rm of type
org.apache.hadoop.yarn.server.resourcemanager.MockRM
[ERROR]
/Users/mingliang.liu/Workspace/apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java:[1741,7]
cannot find symbol
symbol: method
submitApp(int,java.lang.String,java.lang.String,java.util.HashMap,boolean,java.lang.String,int,org.apache.hadoop.security.Credentials)
location: variable rm of type
org.apache.hadoop.yarn.server.resourcemanager.MockRM
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-yarn-server-resourcemanager:
Compilation failure: Compilation failure:
[ERROR]
/Users/mingliang.liu/Workspace/apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java:[1675,7]
cannot find symbol
[ERROR] symbol: method
submitApp(int,java.lang.String,java.lang.String,java.util.HashMap,boolean,java.lang.String,int,org.apache.hadoop.security.Credentials)
[ERROR] location: variable rm of type
org.apache.hadoop.yarn.server.resourcemanager.MockRM
[ERROR]
/Users/mingliang.liu/Workspace/apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java:[1741,7]
cannot find symbol
[ERROR] symbol: method
submitApp(int,java.lang.String,java.lang.String,java.util.HashMap,boolean,java.lang.String,int,org.apache.hadoop.security.Credentials)
[ERROR] location: variable rm of type
org.apache.hadoop.yarn.server.resourcemanager.MockRM
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-yarn-server-resourcemanager
{quote}
Is this broken by this commit? [~elgoiri] and [~maniraj...@gmail.com] Thanks
> RM Renew Delegation token thread should timeout and retry
> -
>
> Key: YARN-9768
> URL: https://issues.apache.org/jira/browse/YARN-9768
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: CR Hota
>Assignee: Manikandan R
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9768.001.patch, YARN-9768.002.patch,
> YARN-9768.003.patch, YARN-9768.004.patch, YARN-9768.005.patch,
> YARN-9768.006.patch, YARN-9768.007.patch, YARN-9768.008.patch
>
>
> Delegation token renewer thread in RM (DelegationTokenRenewer.java) renews
> HDFS tokens received to check for validity and expiration time.
> This call is made to an underlying HDFS NN or Router Node (which has exact
> APIs as HDFS NN). If one of the nodes is bad and the renew call is stuck the
> thread remains stuck indefinitely. The thread should ideally timeout the
> renewToken and retry from the client's perspective.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6239) Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs
[
https://issues.apache.org/jira/browse/YARN-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886643#comment-15886643
]
Mingliang Liu commented on YARN-6239:
-
+1
failing UT is not related.
> Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs
> --
>
> Key: YARN-6239
> URL: https://issues.apache.org/jira/browse/YARN-6239
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: YARN-6239.001.patch
>
>
> There are some javadoc warnings generated after some FileSystem APIs which
> promote inefficient call patterns being deprecated in HADOOP-13321. The
> relevant warnings:
> {code}
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[283,23]
> [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[294,28]
> [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:
> {code}
> This issue is part of HADOOP-14106.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5649) Add REST endpoints for updating application timeouts
[
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691854#comment-15691854
]
Mingliang Liu commented on YARN-5649:
-
{{branch-2}} is broken because of this?
> Add REST endpoints for updating application timeouts
>
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.8.0
>
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch,
> YARN-5649.v0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5833) Add validation to ensure default ports are unique in Configuration
[
https://issues.apache.org/jira/browse/YARN-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649411#comment-15649411
]
Mingliang Liu commented on YARN-5833:
-
Thanks for the prompt action. The {{branch-2}} is great again.
> Add validation to ensure default ports are unique in Configuration
> --
>
> Key: YARN-5833
> URL: https://issues.apache.org/jira/browse/YARN-5833
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5833.003.addendum.patch, YARN-5833.003.patch,
> YARN-5883.001.patch, YARN-5883.002.patch
>
>
> The default port for the AMRMProxy coincides with the one for the Collector
> Service (port 8048). Will use a different port for the AMRMProxy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5833) Add validation to ensure default ports are unique in Configuration
[
https://issues.apache.org/jira/browse/YARN-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649285#comment-15649285
]
Mingliang Liu commented on YARN-5833:
-
{code}
[ERROR] COMPILATION ERROR :
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java:[755,31]
cannot find symbol
symbol: method putIfAbsent(java.lang.String,java.lang.String)
location: variable filteredValues of type
java.util.Map
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-common: Compilation failure
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java:[755,31]
cannot find symbol
[ERROR] symbol: method putIfAbsent(java.lang.String,java.lang.String)
[ERROR] location: variable filteredValues of type
java.util.Map
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-common
{code}
Java 7 on {{branch-2}}. Thanks,
> Add validation to ensure default ports are unique in Configuration
> --
>
> Key: YARN-5833
> URL: https://issues.apache.org/jira/browse/YARN-5833
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5833.003.patch, YARN-5883.001.patch,
> YARN-5883.002.patch
>
>
> The default port for the AMRMProxy coincides with the one for the Collector
> Service (port 8048). Will use a different port for the AMRMProxy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5833) Add validation to ensure default ports are unique in Configuration
[
https://issues.apache.org/jira/browse/YARN-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649223#comment-15649223
]
Mingliang Liu commented on YARN-5833:
-
Is {{branch-2}} failing with Java 7 because of this? Thanks,
> Add validation to ensure default ports are unique in Configuration
> --
>
> Key: YARN-5833
> URL: https://issues.apache.org/jira/browse/YARN-5833
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5833.003.patch, YARN-5883.001.patch,
> YARN-5883.002.patch
>
>
> The default port for the AMRMProxy coincides with the one for the Collector
> Service (port 8048). Will use a different port for the AMRMProxy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4886) Add HDFS caller context for EntityGroupFSTimelineStore
[ https://issues.apache.org/jira/browse/YARN-4886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229240#comment-15229240 ] Mingliang Liu commented on YARN-4886: - +1 (non-binding) > Add HDFS caller context for EntityGroupFSTimelineStore > -- > > Key: YARN-4886 > URL: https://issues.apache.org/jira/browse/YARN-4886 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4886-trunk.001.patch > > > We need to add a HDFS caller context for the entity group FS storage for > better audit log debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
[
https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092772#comment-15092772
]
Mingliang Liu commented on YARN-4265:
-
Hi [~djp] and [~gtCarrera9],
{quote}
For HDFS, any file update should reflect the modification time to parent
directory. Any exception case here?
{quote}
For creating/moving files in an existing directory, the modification time of
the parent directory will be updated. Refer to [code
here|https://github.com/apache/hadoop/blob/9b8e50b424d060e16c1175b1811e7abc476e2468/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java#L1058-L1058].
However, should this jira consider appending to an existing file, the
modification time of the parent directory will not be updated. Refer to [code
here|https://github.com/apache/hadoop/blob/ab56fcdb1219d03713b408dd3a95d7405635254d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java#L77-L77].
Thanks.
> Provide new timeline plugin storage to support fine-grained entity caching
> --
>
> Key: YARN-4265
> URL: https://issues.apache.org/jira/browse/YARN-4265
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-4265-trunk.001.patch, YARN-4265-trunk.002.patch,
> YARN-4265-trunk.003.patch, YARN-4265-trunk.004.patch,
> YARN-4265.YARN-4234.001.patch, YARN-4265.YARN-4234.002.patch
>
>
> To support the newly proposed APIs in YARN-4234, we need to create a new
> plugin timeline store. The store may have similar behavior as the
> EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id
> granularity, instead of application id granularity. Let's have this storage
> as a standalone one, instead of updating EntityFileTimelineStore, to keep the
> existing store (EntityFileTimelineStore) stable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4349) Support CallerContext in YARN
[
https://issues.apache.org/jira/browse/YARN-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001472#comment-15001472
]
Mingliang Liu commented on YARN-4349:
-
Thanks for working on this, [~leftnoteasy].
Minor comment:
{code}
+ CallerContext context =
+ new CallerContext(new CallerContext.Builder(pbContext.getContext())
+ .setSignature(pbContext.getSignature().toByteArray()));
{code}
I think this code can be simplified as:
{code}
+ CallerContext context =
+ new CallerContext.Builder(pbContext.getContext())
+ .setSignature(pbContext.getSignature().toByteArray()).build();
{code}
Is it a good idea to make the constructor
{{CallerContext#CallerContext(Builder)}} private?
> Support CallerContext in YARN
> -
>
> Key: YARN-4349
> URL: https://issues.apache.org/jira/browse/YARN-4349
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-4349.1.patch
>
>
> More details about CallerContext please refer to description of HDFS-9184.
> From YARN's perspective, we should make following changes:
> - RMAuditLogger logs application's caller context when application submit by
> user
> - Add caller context to application's data in ATS along with application
> creation event
> From MR's perspective:
> - Set AppMaster container's context to YARN's application Id
> - Set Mapper/Reducer containers' context to task attempt id
> Protocol and RPC changes are done in HDFS-9184.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
