Re: [DISCUSS] KIP-465: Add Consolidated Connector Endpoint to Connect REST API
This is a nice improvement, both from an operational standpoint as well as
for testing purposes, as we scale the number of connectors that a connect
cluster can run.
LGTM, thanks for the KIP Dan!
On Fri, May 3, 2019 at 2:50 PM Alex Liu wrote:
> Good question,
>
> `info` is probably the best name for it. The updated output on the wiki
> looks reasonable to me.
>
> Alex
>
> On Fri, May 3, 2019 at 2:24 PM dan wrote:
>
> > thanks. i think this make sense.
> >
> > i'm thinking we should just use repeated queryparams for this, so
> > `?expand=status=config`
> >
> > another thing is what do you think we should use for the `/` endpoint?
> was
> > thinking `?expand=info`
> >
> > output could look like
> >
> > w:kafka norwood$ curl -s '
> > http://localhost:8083/connectors?expand=status=config' | jq
> >
> > {
> >
> > "blah": {
> >
> > "config": {
> >
> > "name": "blah",
> >
> > "config": {
> >
> > "connector.class":
> > "org.apache.kafka.connect.file.FileStreamSourceConnector",
> >
> > "file": "/tmp/lol",
> >
> > "tasks.max": "10",
> >
> > "name": "blah",
> >
> > "topic": "test-topic"
> >
> > },
> >
> > "tasks": [
> >
> > {
> >
> > "connector": "blah",
> >
> > "task": 0
> >
> > }
> >
> > ],
> >
> > "type": "source"
> >
> > },
> >
> > "status": {
> >
> > "name": "blah",
> >
> > "connector": {
> >
> > "state": "RUNNING",
> >
> > "worker_id": "10.200.25.241:8083"
> >
> > },
> >
> > "tasks": [
> >
> > {
> >
> > "id": 0,
> >
> > "state": "RUNNING",
> >
> > "worker_id": "10.200.25.241:8083"
> >
> > }
> >
> > ],
> >
> > "type": "source"
> >
> > }
> >
> > }
> >
> > }
> >
> >
> > will update the wiki with this info
> >
> > thanks
> > dan
> >
> > On Thu, May 2, 2019 at 4:43 PM Alex Liu wrote:
> >
> > > Good idea, Dan. One thing I might suggest is to have the query
> parameters
> > > reflect the fact that there are multiple resources under each
> connector.
> > > There is `connectors//`, `connectors//config`, and
> > > `connectors//status`.
> > > Each of them returns a slightly different set of information, so it
> would
> > > be useful to allow the query parameter be a string instead of a
> > true/false
> > > flag. In this case, `expand=status,config` would specify expanding both
> > the
> > > /status and /config subresources into the response objects.
> > >
> > > Other than this detail, I think this is a useful addition to the
> Connect
> > > REST API.
> > >
> > > Alex
> > >
> >
>
Re: [DISCUSS] KIP-465: Add Consolidated Connector Endpoint to Connect REST API
Good question,
`info` is probably the best name for it. The updated output on the wiki
looks reasonable to me.
Alex
On Fri, May 3, 2019 at 2:24 PM dan wrote:
> thanks. i think this make sense.
>
> i'm thinking we should just use repeated queryparams for this, so
> `?expand=status=config`
>
> another thing is what do you think we should use for the `/` endpoint? was
> thinking `?expand=info`
>
> output could look like
>
> w:kafka norwood$ curl -s '
> http://localhost:8083/connectors?expand=status=config' | jq
>
> {
>
> "blah": {
>
> "config": {
>
> "name": "blah",
>
> "config": {
>
> "connector.class":
> "org.apache.kafka.connect.file.FileStreamSourceConnector",
>
> "file": "/tmp/lol",
>
> "tasks.max": "10",
>
> "name": "blah",
>
> "topic": "test-topic"
>
> },
>
> "tasks": [
>
> {
>
> "connector": "blah",
>
> "task": 0
>
> }
>
> ],
>
> "type": "source"
>
> },
>
> "status": {
>
> "name": "blah",
>
> "connector": {
>
> "state": "RUNNING",
>
> "worker_id": "10.200.25.241:8083"
>
> },
>
> "tasks": [
>
> {
>
> "id": 0,
>
> "state": "RUNNING",
>
> "worker_id": "10.200.25.241:8083"
>
> }
>
> ],
>
> "type": "source"
>
> }
>
> }
>
> }
>
>
> will update the wiki with this info
>
> thanks
> dan
>
> On Thu, May 2, 2019 at 4:43 PM Alex Liu wrote:
>
> > Good idea, Dan. One thing I might suggest is to have the query parameters
> > reflect the fact that there are multiple resources under each connector.
> > There is `connectors//`, `connectors//config`, and
> > `connectors//status`.
> > Each of them returns a slightly different set of information, so it would
> > be useful to allow the query parameter be a string instead of a
> true/false
> > flag. In this case, `expand=status,config` would specify expanding both
> the
> > /status and /config subresources into the response objects.
> >
> > Other than this detail, I think this is a useful addition to the Connect
> > REST API.
> >
> > Alex
> >
>
Re: [DISCUSS] KIP-465: Add Consolidated Connector Endpoint to Connect REST API
thanks. i think this make sense.
i'm thinking we should just use repeated queryparams for this, so
`?expand=status=config`
another thing is what do you think we should use for the `/` endpoint? was
thinking `?expand=info`
output could look like
w:kafka norwood$ curl -s '
http://localhost:8083/connectors?expand=status=config' | jq
{
"blah": {
"config": {
"name": "blah",
"config": {
"connector.class":
"org.apache.kafka.connect.file.FileStreamSourceConnector",
"file": "/tmp/lol",
"tasks.max": "10",
"name": "blah",
"topic": "test-topic"
},
"tasks": [
{
"connector": "blah",
"task": 0
}
],
"type": "source"
},
"status": {
"name": "blah",
"connector": {
"state": "RUNNING",
"worker_id": "10.200.25.241:8083"
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
"worker_id": "10.200.25.241:8083"
}
],
"type": "source"
}
}
}
will update the wiki with this info
thanks
dan
On Thu, May 2, 2019 at 4:43 PM Alex Liu wrote:
> Good idea, Dan. One thing I might suggest is to have the query parameters
> reflect the fact that there are multiple resources under each connector.
> There is `connectors//`, `connectors//config`, and
> `connectors//status`.
> Each of them returns a slightly different set of information, so it would
> be useful to allow the query parameter be a string instead of a true/false
> flag. In this case, `expand=status,config` would specify expanding both the
> /status and /config subresources into the response objects.
>
> Other than this detail, I think this is a useful addition to the Connect
> REST API.
>
> Alex
>
RE: [DISCUSS] KIP-465: Add Consolidated Connector Endpoint to Connect REST API
Good idea, Dan. One thing I might suggest is to have the query parameters reflect the fact that there are multiple resources under each connector. There is `connectors//`, `connectors//config`, and `connectors//status`. Each of them returns a slightly different set of information, so it would be useful to allow the query parameter be a string instead of a true/false flag. In this case, `expand=status,config` would specify expanding both the /status and /config subresources into the response objects. Other than this detail, I think this is a useful addition to the Connect REST API. Alex
Re: [DISCUSS] KIP-465: Add Consolidated Connector Endpoint to Connect REST API
Thanks for the KIP, Dan! This is a simple change that will make it easier
to get the status of all of the connectors running in a Connect cluster,
and I like the approach of adding a query parameter to the existing
`connectors/` resource to be backward compatible and to avoid introducing
another resource.
The only minor suggestion I might offer is to highlight that the
ConnectorStateInfo is the same structure returned by the existing
`connectors/{name}/status` resource.
Otherwise, this looks good to me.
Randall
On Wed, May 1, 2019 at 10:39 AM dan wrote:
> The intent of this KIP is to add a consolidate endpoint to connect that
> gives consumers of the api a one stop shop for all their connector info
> needs.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API
>
> thanks,
> dan
>
[DISCUSS] KIP-465: Add Consolidated Connector Endpoint to Connect REST API
The intent of this KIP is to add a consolidate endpoint to connect that gives consumers of the api a one stop shop for all their connector info needs. https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API thanks, dan
