Ferenc Szabo commented on FLUME-2442:

This ticket looked a little bit abandoned, so I would like to provide a 
different solution to the problem.

My approach is different in the way that it handles the passwords in the 
configuration layer without touching the individual components.
This way we do not have to change them and any custom component can use the 
feature without the need to modify them.

The configuration is extended with a new type of component, a configuration 
filter. It has a public API, so users can implement their custom configuration 
filter and we can add new ones later if needed.

I have created 3 implementations for this API:
 - External Process config filter: it is based on a POC created by [~denes] 
 - Hadoop credential store config filter: It uses the Hadoop credential store 
API suggested by [~roshan_naik]
 - Environment variable config filter: It is almost the same as the 
EnvVarResolverProperties by [~bessbd] but it is implemented with the new API so 
it can be used in a consistent way

reviews are more than welcomed

Before adding the new component I have created unit tests for the configuration 
module and did some refactor to make it readable and easier to add the new 
Then added the new API and the implementations with individual unit tests in 
the modules and an end-to-end test in the flume-ng-tests module to test the 

PR is : [https://github.com/apache/flume/pull/197]

> Need an alternative to providing clear text passwords in flume config
> ---------------------------------------------------------------------
>                 Key: FLUME-2442
>                 URL: https://issues.apache.org/jira/browse/FLUME-2442
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions:
>            Reporter: Roshan Naik
>            Assignee: Venkat Ranganathan
>            Priority: Major
>              Labels: Security
>         Attachments: FLUME-2442.patch.7, FLUME-2442.patch.9, 
> FLUME-2442.v1.patch, FLUME-2442.v2.patch, FLUME-2442.v3.patch, 
> FLUME-2442.v4.patch, FLUME-2442.v5.patch
> For some sources and sinks, currently, passwords to keystores/other are 
> specified in clear text in the flume config file.   Since flume config files 
> are often easily accessible to a broader audience (like in source control for 
> instance), the visibility of these passwords can be too much and risky for 
> institutions where security is too critical (like banks) 
> To help address this visibility issue it would be desirable to do the 
> following two things :
> 1) Store the password in a separate file and provide the path of that 
> password file in the flume config. this will enable the flume config to be 
> shared with a wider audience and reduce risk. the password file will need to 
> be very tightly guarded. Some components like file channel & JMS source 
> already do this. 
> 2) As an additional measure, obfuscate the password in the external password 
> file. A simple command line tool can be used to generate the obfuscated 
> password file. Flume source/sink configuration will read the password file 
> and de-obfuscate the password before using it to access the keystore. This 
> obfuscation step IMO is nice but unclear to me if it is essential.
> The following Sources and Sinks appear to use inline cleartext passwords:
> - Avro Source
> - Avro sink
> - HTTP(S) source 
> - File Channel
> - JMS Source
> JDBC channel also uses inline passwords but i am not aware of anybody who 
> uses JDBC channel. So it may not be an issue.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org
For additional commands, e-mail: issues-h...@flume.apache.org

Reply via email to