[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883443#comment-15883443
]
Robert Kanter commented on OOZIE-2803:
--
+1
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882821#comment-15882821
]
Hadoop QA commented on OOZIE-2803:
--
Testing JIRA OOZIE-2803
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 11 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:orange}0{color} There are [1] new bugs found in [sharelib/oozie]
that would be nice to have fixed.
.You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1886
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3658/
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882579#comment-15882579
]
Peter Bacsko commented on OOZIE-2803:
-
There are no findbugs warnings, Raw patch analysis complains about lines that
are too long. But those are just the csv files that contain test data.
I changed {{MapReduceMain}} and {{PasswordMasker}} as you requested.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881715#comment-15881715
]
Robert Kanter commented on OOZIE-2803:
--
Looks good overall. A few trivial things:
- Can you look into the findbugs warning?
- When creating the new {{JobConf}} in {{MapReduceMain}}, I think it would be
better to pass {{false}} to make sure it doesn't load any other properties if
there's some site/default file on the classpath somehow. All of the properties
will be copied in there anyway.
{code:java}
JobConf maskedJobConf = new JobConf(false);
{code}
- Shouldn't {{PASSWORD_EXTRACTING_REGEX}} contain a reference to
{{PASSWORD_KEY}} instead of hardcoding "pass" there? I can't imagine we'd ever
change {{PASSWORD_KEY}}, but that would be more future-proof.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch, OOZIE-2803-004.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878368#comment-15878368
]
Hadoop QA commented on OOZIE-2803:
--
Testing JIRA OOZIE-2803
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 11 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:orange}0{color} There are [1] new bugs found in [sharelib/oozie]
that would be nice to have fixed.
.You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1886
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3654/
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch, OOZIE-2803-004.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878103#comment-15878103
]
Abhishek Bafna commented on OOZIE-2803:
---
[~pbacsko] Thanks for pointing to the previous discussion. I think we can keep
it same.
For the JavaDoc: {{@param String}} and {{@param Input}} there should be
parameter names, which is missing.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878078#comment-15878078
]
Peter Bacsko commented on OOZIE-2803:
-
Thanks for the comment [~abhishekbafna].
1. What's wrong with the JavaDoc? I think it gives an accurate description of
what the methods are doing. Are there spelling or grammatical errors?
2. As for PasswordMasker not being an utility class - I completely agree with
you, originally, [~andras.piros] did not want to write static methods, his
reasoning was the following:
https://issues.apache.org/jira/browse/OOZIE-1814?focusedCommentId=15528763=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15528763
Nevertheless I can make it static, I don't think it's a big deal :)
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877556#comment-15877556
]
Abhishek Bafna commented on OOZIE-2803:
---
Few points:
There are some unused imports in the {{MapReduceMain}}.
Fix the java doc for {{PasswordMasker:maskPasswordsIfNecessary}} and
{{PasswordMasker:mask}}.
All the field members of {{PasswordMasker}} are declared final and class in
itself does not have any state. it can be converted into Utility Class.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876469#comment-15876469
]
Hadoop QA commented on OOZIE-2803:
--
Testing JIRA OOZIE-2803
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 11 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:orange}0{color} There are [1] new bugs found in [sharelib/oozie]
that would be nice to have fixed.
.You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1886
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3653/
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876259#comment-15876259
]
Peter Bacsko commented on OOZIE-2803:
-
Fixing Findbugs problems. The long lines (> 132 chars) occur only in the csv
files, so this warning can be ignored.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876043#comment-15876043
]
Hadoop QA commented on OOZIE-2803:
--
Testing JIRA OOZIE-2803
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 12 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:red}-1{color} There are [2] new bugs found below threshold in
[sharelib/oozie] that must be fixed.
.You can find the FindBugs diff here (look for the red and orange ones):
sharelib/oozie/findbugs-new.html
.The most important FindBugs errors are:
.At PasswordMasker.java:[lines 85-92]: key must be nonnull but is marked as
nullable
.At PasswordMasker.java:[lines 85-92]: value must be nonnull but is marked
as nullable
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1886
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3652/
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874508#comment-15874508
]
Peter Bacsko commented on OOZIE-2803:
-
I changed the approach - after talking with [~andras.piros] and [~gezapeti], I
decided to re-use {{PasswordMasker}}. I had to enhance it a little bit and move
it from {{oozie-core}} to {{sharelib-oozie}} but this solution is better. I
also added extra tests.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871796#comment-15871796
]
Peter Bacsko commented on OOZIE-2803:
-
In MapReduce, we have to mask the following props (if they contain a password):
{{mapred.child.env}}
{{yarn.app.mapreduce.am.env}}
{{oozie.launcher.mapred.child.env}}
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
