Re: [DISCUSSION] KIP-11: ACL Management

Jun Here is the recent interface from Hive. It is grossly simplified from what you pasted before. https://hive.apache.org/javadocs/r1.0.0/api/ql/org/apache/hadoop/hive/ql/se curity/authorization/plugin/HiveAuthorizer.html At the high level, there is only one authorization method (similar to Part

Review Request 33334: Patch for KAFKA-2131

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4/ --- Review request for kafka. Bugs: KAFKA-2131 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-2131) Update new producer javadocs with correct documentation links

[ https://issues.apache.org/jira/browse/KAFKA-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-2131: --- Attachment: KAFKA-2131.patch > Update new producer javadocs with correct documentation links >

[jira] [Updated] (KAFKA-2131) Update new producer javadocs with correct documentation links

[ https://issues.apache.org/jira/browse/KAFKA-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-2131: --- Status: Patch Available (was: Open) > Update new producer javadocs with correct documentation

[jira] [Commented] (KAFKA-2131) Update new producer javadocs with correct documentation links

[ https://issues.apache.org/jira/browse/KAFKA-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501144#comment-14501144 ] Manikumar Reddy commented on KAFKA-2131: Created reviewboard https://reviews.apach

[jira] [Updated] (KAFKA-2131) Update new producer javadocs with correct documentation links

[ https://issues.apache.org/jira/browse/KAFKA-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-2131: --- Description: New producer java docs are referring to old producer documentation. > Update new

[jira] [Created] (KAFKA-2131) Update new producer javadocs with correct documentation links

Manikumar Reddy created KAFKA-2131: -- Summary: Update new producer javadocs with correct documentation links Key: KAFKA-2131 URL: https://issues.apache.org/jira/browse/KAFKA-2131 Project: Kafka

[jira] [Commented] (KAFKA-1884) Print metadata response errors

[ https://issues.apache.org/jira/browse/KAFKA-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501112#comment-14501112 ] Manikumar Reddy commented on KAFKA-1884: [~guozhang] can you review this trivial p

KIP hangout on Apr 21

Hi, We will have a KIP hangout at 3pm PST on Apr 21. The following is the tentative agenda. If you'd like to attend but haven't received an invite, please let me know. Agenda: KIP-4 (admin commands): wrap up any remaining issues KIP-11 (Authorization): KIP-12 (SSL/Kerberos): See if there is any b

[jira] [Commented] (KAFKA-2088) kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect

[ https://issues.apache.org/jira/browse/KAFKA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501055#comment-14501055 ] Zhiqiang He commented on KAFKA-2088: [~junrao] II have read codes but not found where

Re: [DISCUSSION] KIP-11: ACL Management

I looked into the consumer offset storage and it seems like for acl storage we should not need something as complex. Consumer offset has different throughput requirements which is why I think it made sense to move away from zookeeper. Acls on the other hand seldom change and because of the caching

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501020#comment-14501020 ] Sriharsha Chintalapani commented on KAFKA-1690: --- [~junrao] I didn't send my

Build failed in Jenkins: Kafka-trunk #465

See Changes: [junrao] kafka-2119; ConsumerRecord key() and value() methods should not have throws Exception; patched by Ewen Cheslack-Postava; reviewed by Jun Rao -- [...truncated 1249 lines...] kafk

[jira] [Updated] (KAFKA-2119) ConsumerRecord key() and value() methods should not have throws Exception

[ https://issues.apache.org/jira/browse/KAFKA-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2119: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch. +1 and committed to t

[jira] [Updated] (KAFKA-1690) new java producer needs ssl support as a client

[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1690: --- Status: In Progress (was: Patch Available) [~sriharsha], your original patch in KAFKA-1684 only added the SSL

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1928: --- Status: In Progress (was: Patch Available) > Move kafka.network over to using the network classes in > org.a

Re: Review Request 33065: Patch for KAFKA-1928

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/#review80557 --- Took a quick look. Overall, looks good. A few comments below. clie

[jira] [Commented] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500922#comment-14500922 ] Ashish K Singh commented on KAFKA-1982: --- Updated reviewboard https://reviews.apache.

[jira] [Updated] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1982: -- Status: Patch Available (was: In Progress) > change kafka.examples.Producer to use the new java

[jira] [Updated] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1982: -- Attachment: KAFKA-1982_2015-04-17_17:00:40.patch > change kafka.examples.Producer to use the new

Re: Review Request 31369: Patch for KAFKA-1982

> On April 17, 2015, 11:05 p.m., Jun Rao wrote: > > Thanks for the patch. A few more minor comments. Jun thanks for the review again. Addressed your comments. - Ashish --- This is an automatically generated e-mail. To reply, visit: http

Re: Review Request 31369: Patch for KAFKA-1982

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/ --- (Updated April 18, 2015, midnight) Review request for kafka. Bugs: KAFKA-1982

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

Agreed we also need to change in the code of Sender.java to indicate that it resembles REPLICATION_TIMEOUT and not the request Timeout. Thanks, Mayuresh On Thu, Apr 16, 2015 at 1:08 PM, Jiangjie Qin wrote: > Hi Guozhang, > > By implicit timeout for close() and flush(), I meant that currently w

Re: Review Request 33049: Patch for KAFKA-2084

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33049/#review80130 --- clients/src/main/java/org/apache/kafka/common/metrics/MetricConfig.

[jira] [Updated] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1982: --- Status: In Progress (was: Patch Available) > change kafka.examples.Producer to use the new java producer > --

Re: Review Request 31369: Patch for KAFKA-1982

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/#review80555 --- Thanks for the patch. A few more minor comments. clients/src/main/

[jira] [Updated] (KAFKA-2118) Cleaner cannot clean after shutdown during replaceSegments

[ https://issues.apache.org/jira/browse/KAFKA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2118: --- Status: In Progress (was: Patch Available) > Cleaner cannot clean after shutdown during replaceSegments > ---

Re: Review Request 33168: Fix recovery of swap files after broker crash

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33168/#review80543 --- Thanks for the patch. Great finding and very comprehensive tests! Ju

Re: Review Request 31369: Patch for KAFKA-1982

> On March 25, 2015, 4:48 p.m., Mayuresh Gharat wrote: > > Thanks for the review. Addressed your comment. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/#review77747 -

Re: Review Request 31369: Patch for KAFKA-1982

> On April 7, 2015, 10:41 p.m., Jun Rao wrote: > > Sorry for the late review. A few more comments below. Done! - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/#review77523

[jira] [Commented] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500783#comment-14500783 ] Ashish K Singh commented on KAFKA-1982: --- Updated reviewboard https://reviews.apache.

[jira] [Updated] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1982: -- Attachment: KAFKA-1982_2015-04-17_15:08:54.patch > change kafka.examples.Producer to use the new

Re: Review Request 31369: Patch for KAFKA-1982

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/ --- (Updated April 17, 2015, 10:09 p.m.) Review request for kafka. Bugs: KAFKA-19

[jira] [Commented] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500750#comment-14500750 ] Ashish K Singh commented on KAFKA-1982: --- Updated reviewboard https://reviews.apache.

[jira] [Updated] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1982: -- Status: Patch Available (was: In Progress) > change kafka.examples.Producer to use the new java

Re: Review Request 31369: Patch for KAFKA-1982

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/ --- (Updated April 17, 2015, 9:49 p.m.) Review request for kafka. Bugs: KAFKA-198

[jira] [Updated] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1982: -- Attachment: KAFKA-1982_2015-04-17_14:49:34.patch > change kafka.examples.Producer to use the new

[jira] [Updated] (KAFKA-1990) Add unlimited time-based log retention

[ https://issues.apache.org/jira/browse/KAFKA-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1990: --- Status: In Progress (was: Patch Available) > Add unlimited time-based log retention > ---

Re: Review Request 31590: Patch for KAFKA-1990

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31590/#review80526 --- Thanks for the patch. A few comments below. Also, yes, LogConfigTes

Re: [DISCUSSION] KIP-11: ACL Management

I have copied Thejas from hive team in cc list. Here is what I learnt from him * Hive calls the authorizer plugin if you execute “grant/revoke Operation to User on Table". They use this as hive provides the SQL layer and SQL has standards for grant/revoke which they follow. * If the plugin provid

[jira] [Updated] (KAFKA-2130) Resource leakage in AppInfo.scala during initialization

[ https://issues.apache.org/jira/browse/KAFKA-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastien Zimmer updated KAFKA-2130: Attachment: patch.diff > Resource leakage in AppInfo.scala during initialization > -

[jira] [Issue Comment Deleted] (KAFKA-2130) Resource leakage in AppInfo.scala during initialization

[ https://issues.apache.org/jira/browse/KAFKA-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastien Zimmer updated KAFKA-2130: Comment: was deleted (was: diff --git a/core/src/main/scala/kafka/common/AppInfo.scala b/co

[jira] [Updated] (KAFKA-2130) Resource leakage in AppInfo.scala during initialization

[ https://issues.apache.org/jira/browse/KAFKA-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastien Zimmer updated KAFKA-2130: Status: Patch Available (was: Open) diff --git a/core/src/main/scala/kafka/common/AppInfo.s

[jira] [Created] (KAFKA-2130) Resource leakage in AppInfo.scala during initialization

Sebastien Zimmer created KAFKA-2130: --- Summary: Resource leakage in AppInfo.scala during initialization Key: KAFKA-2130 URL: https://issues.apache.org/jira/browse/KAFKA-2130 Project: Kafka I

[jira] [Commented] (KAFKA-2088) kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect

[ https://issues.apache.org/jira/browse/KAFKA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500473#comment-14500473 ] Gwen Shapira commented on KAFKA-2088: - I think the main problem here is that someone m

Re: [DISCUSSION] KIP-11: ACL Management

Hi, Parth, How does this work in Hive? I thought authorization in Hive always goes through it's SQL cli for any authorization plugin. When integrating with Ranger(Argus), does Hive do authorization through a separate CLI? Thanks, Jun On Fri, Apr 17, 2015 at 11:01 AM, Parth Brahmbhatt < pbrahmb

[jira] [Comment Edited] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation

[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500437#comment-14500437 ] Ashish K Singh edited comment on KAFKA-1994 at 4/17/15 6:56 PM:

[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation

[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500437#comment-14500437 ] Ashish K Singh commented on KAFKA-1994: --- [~junrao], [~gwenshap], does this look on n

[jira] [Commented] (KAFKA-2088) kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect

[ https://issues.apache.org/jira/browse/KAFKA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500330#comment-14500330 ] Jun Rao commented on KAFKA-2088: [~ransom], thanks for the patch. It may be reasonable to

Re: [DISCUSSION] KIP-11: ACL Management

We could do this but I think its too simplistic plus now we are adding authorization related options in CLI which I thought everyone wants to avoid. When I say its too simplistic I mean there are missing options like —hosts, what happens when we start supporting group now we will probably end up a

[jira] [Commented] (KAFKA-2088) kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect

[ https://issues.apache.org/jira/browse/KAFKA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500289#comment-14500289 ] Gwen Shapira commented on KAFKA-2088: - Pinging [~junrao] [~jjkoshy] [~nehanarkhede] [~

[jira] [Commented] (KAFKA-2088) kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect

[ https://issues.apache.org/jira/browse/KAFKA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500286#comment-14500286 ] Gwen Shapira commented on KAFKA-2088: - Awesome, thanks for checking :) > kafka-consol

Re: [DISCUSSION] KIP-11: ACL Management

I've probably been a DBA for too long, but I imagined something like: kafka-topic --topic t1 --grant user --action action kafka-topic --topic t1 --revoke user --action action (i.e. the commandline equivalent of "grant select on table1 to gwenshap" and "revoke select on table2 from gwenshap") When

Re: [DISCUSSION] KIP-11: ACL Management

* Yes, Acl pretty much captures everything. Originally I had resource as part of Acls, we can go back to that. * The describe can call getAcl and I plan to do so. addAcl is tricky because the user will have to specify the acls through command lines, which will probably be a location to some file. B

Re: [DISCUSSION] KIP-11: ACL Management

On Fri, Apr 17, 2015 at 9:31 AM, Parth Brahmbhatt wrote: > I was following the storm model but I think this is a reasonable change. I > recommend changing the API names to addAcls, removeAcls and getAcls. And they probably just need to get List instead of everything I specified? Looks like Acl e

Re: [DISCUSSION] KIP-11: ACL Management

I was following the storm model but I think this is a reasonable change. I recommend changing the API names to addAcls, removeAcls and getAcls. Couple of points to ensure we are on same page: * With this approach the kafka command line will not provide a way to add/edit acls during topic creatio

[jira] [Commented] (KAFKA-2088) kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect

[ https://issues.apache.org/jira/browse/KAFKA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500104#comment-14500104 ] Zhiqiang He commented on KAFKA-2088: [~gwenshap] I have aleady test topic command in k

[jira] [Commented] (KAFKA-2088) kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect

[ https://issues.apache.org/jira/browse/KAFKA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500026#comment-14500026 ] Gwen Shapira commented on KAFKA-2088: - Thanks for fixing this and for the friendly err

Re: [DISCUSSION] KIP-11: ACL Management

Gwen, There is one product called ElasticSearch which has been quite successful. They recently added security, what they actually did is quite nice. They really separated Authentication and Authorization which many people get really confused about and often mix them up. I looked through w

Review request for KAFKA-2088

KAFKA-2088 is ready for review, any one can help me review it? thanks. JIRA : https://issues.apache.org/jira/browse/KAFKA-2088 review request : https://reviews.apache.org/r/33132/diff/ ransom

Re: [DISCUSSION] KIP-11: ACL Management

On Thu, Apr 16, 2015 at 6:13 PM, Jun Rao wrote: > Hi, Gwen, > > What you suggested seems reasonable. I guess we will need the Privilege> pair and the Resource in grant() and revoke()? I thought that Privilege is a Resource+Action, which is why grant and revoke can take list of principals and lis

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499210#comment-14499210 ] Suresh Srinivas commented on KAFKA-2120: [~ewencp], thanks for the explanation.

Re: Review Request 32937: Patch for KAFKA-2102

> On April 16, 2015, 5:34 a.m., Ewen Cheslack-Postava wrote: > > clients/src/main/java/org/apache/kafka/clients/MetadataBookkeeper.java, > > line 25 > > > > > > This patch will definitely need a comment somewhere expla

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499093#comment-14499093 ] Ewen Cheslack-Postava commented on KAFKA-2120: -- [~sureshms] They're related,