[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830783#comment-16830783
 ] 

ASF GitHub Bot commented on KAFKA-5117:
---

rhauch commented on pull request #4441: [KAFKA-5117]: Password Mask to Kafka 
Connect REST Endpoint
URL: https://github.com/apache/kafka/pull/4441
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>Assignee: Chris Egerton
>Priority: Major
>  Labels: needs-kip
> Fix For: 2.2.0, 2.1.1, 2.0.2
>
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750334#comment-16750334
 ] 

ASF GitHub Bot commented on KAFKA-5117:
---

ewencp commented on pull request #6129: KAFKA-5117: Stop resolving externalized 
configs in Connect REST API
URL: https://github.com/apache/kafka/pull/6129
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>Assignee: Randall Hauch
>Priority: Major
>  Labels: needs-kip
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687038#comment-16687038
 ] 

ASF GitHub Bot commented on KAFKA-5117:
---

qiao-meng-zefr closed pull request #4269: KAFKA-5117: Add password masking for 
kafka connect REST endpoint
URL: https://github.com/apache/kafka/pull/4269
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java
 
b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java
index fbe0ae2afb2..9eb2def0c64 100644
--- 
a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java
+++ 
b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java
@@ -184,6 +184,22 @@ public void resumeConnector(String connector) {
 configBackingStore.putTargetState(connector, TargetState.STARTED);
 }
 
+@Override
+public Map maskCredentials(String connName, Map config) {
+Map newConfig = new LinkedHashMap<>();
+for (Map.Entry entry : config.entrySet()) {
+// Password.toString() will return the hidden value
+String value = null;
+if (entry.getValue() != null) {
+value = entry.getValue().toString();
+}
+
+newConfig.put(entry.getKey(), value);
+}
+
+return newConfig;
+}
+
 @Override
 public Plugins plugins() {
 return worker.getPlugins();
diff --git 
a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Herder.java 
b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Herder.java
index 5dfb808f764..774dbaf8ca1 100644
--- a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Herder.java
+++ b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Herder.java
@@ -169,6 +169,14 @@
  */
 void resumeConnector(String connector);
 
+/**
+ * Goes through config parameters and replace password field value with 
"[hidden"]
+ * @param connName name of the connector
+ * @param config configuration of the connector
+ * @return new map of the configurations, with password omitted from 
clear-text
+ */
+Map maskCredentials(String connName, Map 
config);
+
 /**
  * Returns a handle to the plugin factory used by this herder and its 
worker.
  *
diff --git 
a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/resources/ConnectorsResource.java
 
b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/resources/ConnectorsResource.java
index 2c031245c06..6e19cc729a8 100644
--- 
a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/resources/ConnectorsResource.java
+++ 
b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/resources/ConnectorsResource.java
@@ -50,6 +50,7 @@
 import java.util.Collection;
 import java.util.List;
 import java.util.Map;
+import java.util.ArrayList;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
@@ -109,7 +110,8 @@ public ConnectorInfo getConnector(final 
@PathParam("connector") String connector
   final @QueryParam("forward") Boolean 
forward) throws Throwable {
 FutureCallback cb = new FutureCallback<>();
 herder.connectorInfo(connector, cb);
-return completeOrForwardRequest(cb, "/connectors/" + connector, "GET", 
null, forward);
+ConnectorInfo connectorInfo = completeOrForwardRequest(cb, 
"/connectors/" + connector, "GET", null, forward);
+return new ConnectorInfo(connectorInfo.name(), 
herder.maskCredentials(connector, connectorInfo.config()), 
connectorInfo.tasks(), connectorInfo.type());
 }
 
 @GET
@@ -118,7 +120,8 @@ public ConnectorInfo getConnector(final 
@PathParam("connector") String connector
   final @QueryParam("forward") 
Boolean forward) throws Throwable {
 FutureCallback> cb = new FutureCallback<>();
 herder.connectorConfig(connector, cb);
-return completeOrForwardRequest(cb, "/connectors/" + connector + 
"/config", "GET", null, forward);
+Map config = completeOrForwardRequest(cb, 
"/connectors/" + connector + "/config", "GET", null, forward);
+return herder.maskCredentials(connector, config);
 }
 
 @GET
@@ -177,8 +180,14 @@ public Response resumeConnector(@PathParam("connector") 
String connector) {
  final @QueryParam("forward") Boolean 
forward) throws Throwable {
 FutureCallback> 

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640116#comment-16640116
 ] 

Ewen Cheslack-Postava commented on KAFKA-5117:
--

Going to close this since 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations]
 addresses this problem. Feel free to reopen if that doesn't sufficiently 
address the issue.

> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>Priority: Major
>  Labels: needs-kip
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[satyanarayan komandur (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606326#comment-16606326
 ] 

satyanarayan komandur commented on KAFKA-5117:
--

I would like to add couple of more points related to this KIP

Currently i noticed even accessing end point

connectors/\{connector-name}/status is also hitting the configuration. I think 
this endpoint need not gather config information.

 

 

 

> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>Priority: Major
>  Labels: needs-kip
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331104#comment-16331104
 ] 

ASF GitHub Bot commented on KAFKA-5117:
---

Tang8330 opened a new pull request #4441: [KAFKA-5117]: Password Mask to Kafka 
Connect REST Endpoint
URL: https://github.com/apache/kafka/pull/4441
 
 
   ⚠️  WIP ⚠️ 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>Priority: Major
>  Labels: needs-kip
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324215#comment-16324215
 ] 

Randall Hauch commented on KAFKA-5117:
--

So I'm a bit concerned that simply masking the passwords will not be that 
advantageous. Sure, it might work if you're just managing your configuration 
files locally and then using the REST API with curl and thus never really 
needing to get configurations back out. But this change would likely break 
every management tool that is using the API to read, modify, and post 
configurations. Also, to maintain backward compatibility, we'd need to 
introduce a config file that defaults to _not masking_ -- doesn't that kind of 
defeat the purpose?

[KIP-208|https://cwiki.apache.org/confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface]
 is already trying to add SSL/TLS support to the Connect REST API, and then 
adding (with a different KIP) ACLs support would mean you can control who can 
and cannot use different endpoints. That is definitely one approach to 
preventing exposure of passwords.

Another approach is to avoid putting passwords in the configuration file in the 
first place. KAFKA-6142 proposes adding support for variables in configuration 
files, and variables could be used in place of passwords to have the passwords 
resolved only upon deployment via some "configuration transformer" plugin.

> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>  Labels: needs-kip
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286021#comment-16286021
 ] 

Randall Hauch commented on KAFKA-5117:
--

BTW, the KIP doesn't have to be that complex, since this is a straighforward 
change. Just follow the process outlined 
[here|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals].
 The "Migration Plan and Compatibility" section of the KIP should highlight the 
fact that the public response of several methods will change to mask the 
password configuration values.

> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>  Labels: needs-kip
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286014#comment-16286014
 ] 

Randall Hauch commented on KAFKA-5117:
--

This issue is requesting a change in a public API for several Connect REST 
methods, and therefore we're going to need a KIP for this change to make sure 
that all considerations are taken into account.

> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>  Labels: needs-kip
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267400#comment-16267400
 ] 

ASF GitHub Bot commented on KAFKA-5117:
---

GitHub user qiao-meng-zefr opened a pull request:

https://github.com/apache/kafka/pull/4269

KAFKA-5117: Add password masking for kafka connect REST endpoint

*More detailed description of your change,
Mask all password type config parameter with "*" instead of displaying the 
plain text in kafka connect REST endpoint.

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/qiao-meng-zefr/kafka mask_password

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4269.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4269


commit 51665eae52eadb9e6db4107dffac12e0bd40585a
Author: Vincent Meng 
Date:   2017-11-27T19:41:11Z

Add password masking




> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)