[jira] [Commented] (MAPREDUCE-7141) Allow KMS generated spill encryption keys
[
https://issues.apache.org/jira/browse/MAPREDUCE-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739792#comment-16739792
]
Kuhu Shukla commented on MAPREDUCE-7141:
This patch is not perfect and has one big challenge. How do we decouple
ourselves for KMSSpillKeyProvider to not use the DistributedFileSystem for
getting the key provider shown below. What does it take for the RM to provide
an independent key provider.
{code}KeyProvider keyProvider =
((DistributedFileSystem)FileSystem.get(getConf())).getKeyProvider();
{code}
I appreciate any and all comments on the same.
> Allow KMS generated spill encryption keys
> -
>
> Key: MAPREDUCE-7141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7141
> Project: Hadoop Map/Reduce
> Issue Type: Task
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Major
> Attachments: MAPREDUCE-7141-001.patch
>
>
> Currently the only way an encryption key for task spills is generated is by
> the AM's key generator. This JIRA tracks the work required to add KMS support
> to this key's generation allowing fault tolerance to AM failures/re-runs and
> also give another option to the client on how it wants the keys to be created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7141) Allow KMS generated spill encryption keys
[ https://issues.apache.org/jira/browse/MAPREDUCE-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7141 started by Kuhu Shukla. -- > Allow KMS generated spill encryption keys > - > > Key: MAPREDUCE-7141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7141 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: MAPREDUCE-7141-001.patch > > > Currently the only way an encryption key for task spills is generated is by > the AM's key generator. This JIRA tracks the work required to add KMS support > to this key's generation allowing fault tolerance to AM failures/re-runs and > also give another option to the client on how it wants the keys to be created. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7141) Allow KMS generated spill encryption keys
[ https://issues.apache.org/jira/browse/MAPREDUCE-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated MAPREDUCE-7141: --- Attachment: MAPREDUCE-7141-001.patch > Allow KMS generated spill encryption keys > - > > Key: MAPREDUCE-7141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7141 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: MAPREDUCE-7141-001.patch > > > Currently the only way an encryption key for task spills is generated is by > the AM's key generator. This JIRA tracks the work required to add KMS support > to this key's generation allowing fault tolerance to AM failures/re-runs and > also give another option to the client on how it wants the keys to be created. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6709) Add configurable flag to allow MapReduce AM to specify the 'ensureExecutionType' of a Request
[
https://issues.apache.org/jira/browse/MAPREDUCE-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739651#comment-16739651
]
Hadoop QA commented on MAPREDUCE-6709:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
12s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
18s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 28s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
56s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}239m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | MAPREDUCE-6709 |
| JIRA
[jira] [Updated] (MAPREDUCE-6709) Add configurable flag to allow MapReduce AM to specify the 'ensureExecutionType' of a Request
[ https://issues.apache.org/jira/browse/MAPREDUCE-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated MAPREDUCE-6709: - Attachment: MAPREDUCE-6709.002.patch > Add configurable flag to allow MapReduce AM to specify the > 'ensureExecutionType' of a Request > - > > Key: MAPREDUCE-6709 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6709 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Abhishek Modi >Priority: Major > Attachments: MAPREDUCE-6709.001.patch, MAPREDUCE-6709.002.patch > > > MAPREDUCE-6703 allows users to configure the ratio of Map tasks to be > requested as OPPORTUNISTIC. This JIRA proposes to expose configuration to > allow users to additionally specify the value *ensureExecutionType* flag > (introduced in YARN-5180) as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
