Re: Mirroring multiple clusters into one

2017-07-06 Thread James Cheng
Answers inline below. -James Sent from my iPhone > On Jul 7, 2017, at 1:18 AM, Vahid S Hashemian > wrote: > > James, > > Thanks for sharing your thoughts and experience. > Could you please also confirm whether > - you do any encryption for the mirrored data? Not

[GitHub] kafka pull request #3499: KAFKA-5567: Connect sink worker should commit offs...

2017-07-06 Thread kkonstantine
GitHub user kkonstantine opened a pull request: https://github.com/apache/kafka/pull/3499 KAFKA-5567: Connect sink worker should commit offsets of original topic partitions You can merge this pull request into a Git repository by running: $ git pull

Re: Dealing with noisy timestamps in kafka streams + 10.1

2017-07-06 Thread Greg Fodor
I managed to answer some of my own questions :) For future google'ers: to deal with retention.ms see https://issues.apache.org/jira/browse/KAFKA-4340 to deal with early rejection of bad timestamps the message.timestamp.difference.max.ms config is relevant discussion here

Fwd: How to fetch offset for all the partitions of one topic through single KafkaClient function call (0.10.x)?

2017-07-06 Thread Martin Peng
Hi, I am trying to use Apache Kafka client 0.10.2 to do batch offset fetching for all partitions of one topic. However I found the KafkaConsumer.commited() only provided the ability to fetch offset for single partition. I read the KafkaConsumer class code and found actually it does have the

[GitHub] kafka pull request #3498: Kafka-4763 (Used for triggering test only)

2017-07-06 Thread lindong28
Github user lindong28 closed the pull request at: https://github.com/apache/kafka/pull/3498 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3498: Kafka-4763 (Used for triggering test only)

2017-07-06 Thread lindong28
GitHub user lindong28 reopened a pull request: https://github.com/apache/kafka/pull/3498 Kafka-4763 (Used for triggering test only) This patch is used only for triggering test. No need for review. You can merge this pull request into a Git repository by running: $ git pull

[jira] [Created] (KAFKA-5568) Transformations that mutate topic-partitions break sink connectors that manage their own configuration

2017-07-06 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-5568: Summary: Transformations that mutate topic-partitions break sink connectors that manage their own configuration Key: KAFKA-5568 URL:

[jira] [Created] (KAFKA-5567) With transformations that mutate the topic-partition committing offsets should to refer to the original topic-partition

2017-07-06 Thread Konstantine Karantasis (JIRA)
Konstantine Karantasis created KAFKA-5567: - Summary: With transformations that mutate the topic-partition committing offsets should to refer to the original topic-partition Key: KAFKA-5567 URL:

[GitHub] kafka-site pull request #66: New landing page for Streams API

2017-07-06 Thread derrickdoo
GitHub user derrickdoo opened a pull request: https://github.com/apache/kafka-site/pull/66 New landing page for Streams API New Kafka Streams API landing page for desktop and mobile web. Note: There are a couple of items that need to be filled in before we launch this

Re: Mirroring multiple clusters into one

2017-07-06 Thread Vahid S Hashemian
James, Thanks for sharing your thoughts and experience. Could you please also confirm whether - you do any encryption for the mirrored data? - you have a many-to-one mirroring similar to what I described? Thanks. --Vahid From: James Cheng To:

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-07-06 Thread Jeyhun Karimov
Hi Damian, Thanks for comments. About overrides, what other alternatives do we have? For backwards-compatibility we have to add extra methods to the existing ones. About ProcessorContext vs RecordContext, you are right. I think I need to implement a prototype to understand the full picture as

[jira] [Resolved] (KAFKA-4726) ValueMapper should have (read) access to key

2017-07-06 Thread Matthias J. Sax (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-4726. Resolution: Duplicate See

[jira] [Resolved] (KAFKA-3745) Consider adding join key to ValueJoiner interface

2017-07-06 Thread Matthias J. Sax (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-3745. Resolution: Duplicate See

[jira] [Created] (KAFKA-5566) Instable test QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied

2017-07-06 Thread Matthias J. Sax (JIRA)
Matthias J. Sax created KAFKA-5566: -- Summary: Instable test QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied Key: KAFKA-5566 URL: https://issues.apache.org/jira/browse/KAFKA-5566

[GitHub] kafka pull request #3498: Kafka-4763 (Used for triggering test only)

2017-07-06 Thread lindong28
Github user lindong28 closed the pull request at: https://github.com/apache/kafka/pull/3498 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3498: Kafka-4763 (Used for triggering test only)

2017-07-06 Thread lindong28
GitHub user lindong28 reopened a pull request: https://github.com/apache/kafka/pull/3498 Kafka-4763 (Used for triggering test only) This patch is used only for triggering test. No need for review. You can merge this pull request into a Git repository by running: $ git pull

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-06 Thread Greg Fodor
Also sorry, to clarify the job context: - This is a job running across 5 nodes on AWS Linux - It is under load with a large number of partitions: approximately 700-800 topic-partitions assignments in total for the entire job. Topics involved have large # of partitions, 128 each. - 32 stream

Re: Wiki access

2017-07-06 Thread Jun Rao
Hi, Mitch, Thanks for your interest. Just gave you the wiki permission. Jun On Thu, Jul 6, 2017 at 10:43 AM, Mitch Seymour wrote: > Hello, > > I'd like to make some contributions to the wiki. Could someone please grant > write permissions to the following user? > >

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-06 Thread Jeff Widman
Thanks for the KIP Vahid. I think it'd be useful to have these filters. That said, I also agree with Edo. We don't currently rely on the output, but there's been more than one time when debugging an issue that I notice something amiss when I see all the data at once but if it wasn't present in

[GitHub] kafka pull request #3498: Kafka-4763 (Used for triggering test only)

2017-07-06 Thread lindong28
Github user lindong28 closed the pull request at: https://github.com/apache/kafka/pull/3498 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3498: Kafka-4763 (Used for triggering test only)

2017-07-06 Thread lindong28
GitHub user lindong28 reopened a pull request: https://github.com/apache/kafka/pull/3498 Kafka-4763 (Used for triggering test only) This patch is used only for triggering test. No need for review. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3498: Kafka-4763 (Used for triggering test only)

2017-07-06 Thread lindong28
GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/3498 Kafka-4763 (Used for triggering test only) This patch is used only for triggering test. No need for review. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3497: HOTFIX: disable flaky system tests

2017-07-06 Thread mjsax
GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/3497 HOTFIX: disable flaky system tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka disable-flaky-system-tests Alternatively you can

Re: [VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-07-06 Thread Gwen Shapira
+1 On Wed, Jul 5, 2017 at 9:25 AM Matthias J. Sax wrote: > +1 > > On 6/27/17 1:41 PM, Jeyhun Karimov wrote: > > Dear all, > > > > I would like to start the vote on KIP-149 [1]. > > > > > > Cheers, > > Jeyhun > > > > > > [1] > > >

Re: Mirroring multiple clusters into one

2017-07-06 Thread James Cheng
I'm not sure what the "official" recommendation is. At TiVo, we *do* run all our mirrormakers near the target cluster. It works fine for us, but we're still fairly inexperienced, so I'm not sure how strong of a data point we should be. I think the thought process is, if you are mirroring from a

Dealing with noisy timestamps in kafka streams + 10.1

2017-07-06 Thread Greg Fodor
Hey all, we are currently working on migrating our system to kafka 10.2 from 10.0 and one thing that we have hit that I wanted some advice on is dealing with the new log retention/rolling semantics that are based on timestamps. We send telemetry data from installed clients into kafka via kafka

[jira] [Created] (KAFKA-5565) Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-06 Thread Colin P. McCabe (JIRA)
Colin P. McCabe created KAFKA-5565: -- Summary: Add a broker metric specifying the number of consumer group rebalances in progress Key: KAFKA-5565 URL: https://issues.apache.org/jira/browse/KAFKA-5565

Re: [DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

2017-07-06 Thread Gwen Shapira
This sounds great. I'll try to review later today :) On Thu, Jul 6, 2017 at 12:35 AM Sönke Liebau wrote: > I've updated the pull request to behave as follows: > - reject create requests that contain no "name" element with a > BadRequestException > - reject

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-06 Thread Greg Fodor
That's great news, thanks! On Thu, Jul 6, 2017 at 6:18 AM, Damian Guy wrote: > Hi Greg, > I've been able to reproduce it by running multiple instances with standby > tasks and many threads. If i force some rebalances, then i see the failure. > Now to see if i can repro in

Wiki access

2017-07-06 Thread Mitch Seymour
Hello, I'd like to make some contributions to the wiki. Could someone please grant write permissions to the following user? Username: mitch-seymour Thanks, Mitch

[GitHub] kafka-site issue #65: MINOR: add Interactive Queries docs

2017-07-06 Thread guozhangwang
Github user guozhangwang commented on the issue: https://github.com/apache/kafka-site/pull/65 Actually I think we do not need a PR for this: as a committer anyone can do the docs jar packaging -> pasting to `kafka-site` and push to `asf-site` branch. You can follow this step of the

[GitHub] kafka-site issue #64: MINOR: Add release dates to downloads page

2017-07-06 Thread ewencp
Github user ewencp commented on the issue: https://github.com/apache/kafka-site/pull/64 I grabbed the dates from that page and filled in the beta ones based on mailing list posts. The actual dates seem to be inconsistent and the JIRA release dates aren't always maintained (I know

[GitHub] kafka pull request #3496: KAFKA-5464: Follow up. Increase poll timeout

2017-07-06 Thread mjsax
GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/3496 KAFKA-5464: Follow up. Increase poll timeout You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka KAFKA-5464-follow-up Alternatively you

Jenkins build is back to normal : kafka-trunk-jdk8 #1789

2017-07-06 Thread Apache Jenkins Server
See

Build failed in Jenkins: kafka-trunk-jdk7 #2498

2017-07-06 Thread Apache Jenkins Server
See Changes: [damian.guy] HOTFIX: fix broken streams test -- [...truncated 2.45 MB...] org.apache.kafka.streams.processor.internals.ProcessorTopologyTest >

[GitHub] kafka pull request #3493: HOTFIX: fix broken streams test

2017-07-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3493 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-06 Thread Damian Guy
Hi Greg, I've been able to reproduce it by running multiple instances with standby tasks and many threads. If i force some rebalances, then i see the failure. Now to see if i can repro in a test. I think it is probably the same issue as: https://issues.apache.org/jira/browse/KAFKA-5070 On Thu, 6

[jira] [Created] (KAFKA-5564) Fail to create topics with error 'While recording the replica LEO, the partition [topic2,0] hasn't been created'

2017-07-06 Thread Klearchos Chaloulos (JIRA)
Klearchos Chaloulos created KAFKA-5564: -- Summary: Fail to create topics with error 'While recording the replica LEO, the partition [topic2,0] hasn't been created' Key: KAFKA-5564 URL:

[GitHub] kafka pull request #3495: KAFKA-3362: Update protocol schema and field doc s...

2017-07-06 Thread andrasbeni
GitHub user andrasbeni opened a pull request: https://github.com/apache/kafka/pull/3495 KAFKA-3362: Update protocol schema and field doc strings Added doc to protocol fields You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3494: MINOR: Move quickstart under streams

2017-07-06 Thread enothereska
GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/3494 MINOR: Move quickstart under streams You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka minor-quickstart-docs Alternatively

Build failed in Jenkins: kafka-trunk-jdk8 #1788

2017-07-06 Thread Apache Jenkins Server
See Changes: [damian.guy] KAFKA-5372; fixes to state transitions [damian.guy] KAFKA-5508; Documentation for altering topics -- [...truncated 2.46 MB...]

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-06 Thread Damian Guy
Greg, what OS are you running on? Are you able to reproduce this in a test at all? For instance, based on what you described it would seem that i should be able to start a streams app, wait for it to be up and running, run the state dir cleanup, see it fail. However, i can't reproduce it. On Wed,

[GitHub] kafka pull request #3492: HOTFIX: Hotfix for trunk failing

2017-07-06 Thread enothereska
Github user enothereska closed the pull request at: https://github.com/apache/kafka/pull/3492 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

Jenkins build is back to normal : kafka-0.11.0-jdk7 #210

2017-07-06 Thread Apache Jenkins Server
See

[GitHub] kafka pull request #3493: HOTFIX: fix broken streams test

2017-07-06 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3493 HOTFIX: fix broken streams test You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka hotfix-test-failure Alternatively you can review and

Build failed in Jenkins: kafka-trunk-jdk7 #2497

2017-07-06 Thread Apache Jenkins Server
See Changes: [damian.guy] KAFKA-5372; fixes to state transitions [damian.guy] KAFKA-5508; Documentation for altering topics -- [...truncated 2.46 MB...]

[GitHub] kafka pull request #3492: HOTFIX: Hotfix for trunk failing

2017-07-06 Thread enothereska
GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/3492 HOTFIX: Hotfix for trunk failing Disable 3 tests that need rethinking. You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka

[jira] [Created] (KAFKA-5563) Clarify handling of connector name in config

2017-07-06 Thread JIRA
Sönke Liebau created KAFKA-5563: --- Summary: Clarify handling of connector name in config Key: KAFKA-5563 URL: https://issues.apache.org/jira/browse/KAFKA-5563 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-5562) Do streams state directory cleanup on a single thread

2017-07-06 Thread Damian Guy (JIRA)
Damian Guy created KAFKA-5562: - Summary: Do streams state directory cleanup on a single thread Key: KAFKA-5562 URL: https://issues.apache.org/jira/browse/KAFKA-5562 Project: Kafka Issue Type:

Build failed in Jenkins: kafka-trunk-jdk7 #2496

2017-07-06 Thread Apache Jenkins Server
See Changes: [damian.guy] KAFKA-5464; StreamsKafkaClient should not use -- [...truncated 2.45 MB...] org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest

[jira] [Created] (KAFKA-5561) Rewrite TopicCommand using the new Admin client

2017-07-06 Thread Paolo Patierno (JIRA)
Paolo Patierno created KAFKA-5561: - Summary: Rewrite TopicCommand using the new Admin client Key: KAFKA-5561 URL: https://issues.apache.org/jira/browse/KAFKA-5561 Project: Kafka Issue Type:

Build failed in Jenkins: kafka-0.11.0-jdk7 #209

2017-07-06 Thread Apache Jenkins Server
See Changes: [damian.guy] KAFKA-5464; StreamsKafkaClient should not use -- [...truncated 969.68 KB...] kafka.producer.AsyncProducerTest > testBatchSize PASSED

Build failed in Jenkins: kafka-0.10.2-jdk7 #180

2017-07-06 Thread Apache Jenkins Server
See Changes: [damian.guy] KAFKA-5464; StreamsKafkaClient should not use -- [...truncated 160.53 KB...] kafka.server.AbstractFetcherThreadTest >

[GitHub] kafka pull request #3429: KAFKA-5508: Documentation for altering topics

2017-07-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3429 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5508) Documentation for altering topics

2017-07-06 Thread Damian Guy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5508. --- Resolution: Fixed Fix Version/s: 0.11.0.1 0.11.1.0 Issue resolved by pull

[GitHub] kafka pull request #3432: KAFKA-5372: fixes to state transitions

2017-07-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3432 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3439: KAFKA-5464: StreamsKafkaClient should not use Stre...

2017-07-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3439 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

2017-07-06 Thread Ismael Juma
Thanks everyone! Ismael On Wed, Jul 5, 2017 at 9:55 PM, Jun Rao wrote: > Hi, Everyone, > > Ismael Juma has been active in the Kafka community since he became > a Kafka committer about a year ago. I am glad to announce that Ismael is > now a member of Kafka PMC. > >

[GitHub] kafka-site pull request #65: MINOR: add Interactive Queries docs

2017-07-06 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka-site/pull/65 MINOR: add Interactive Queries docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka-site iq-doc-update Alternatively you can review

[GitHub] kafka-site issue #65: MINOR: add Interactive Queries docs

2017-07-06 Thread dguy
Github user dguy commented on the issue: https://github.com/apache/kafka-site/pull/65 @guozhangwang --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] kafka pull request #3491: HOTFIX: Fixes to metric names

2017-07-06 Thread enothereska
GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/3491 HOTFIX: Fixes to metric names A couple of fixes to metric names to match the KIP - Removed extra strings in the metric names that are already in the tags - add a separate metric for

Re: 答复: [ANNOUNCE] New Kafka PMC member Ismael Juma

2017-07-06 Thread Paolo Patierno
Congratulations Ismael ... well deserved ! Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoT Microsoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog :

Re: 答复: [ANNOUNCE] New Kafka PMC member Ismael Juma

2017-07-06 Thread Mickael Maison
Congratulations Ismael ! On Thu, Jul 6, 2017 at 8:57 AM, Hu Xi wrote: > Congrats, Ismael  > > > > 发件人: Molnár Bálint > 发送时间: 2017年7月6日 15:51 > 收件人: dev@kafka.apache.org > 主题: Re: [ANNOUNCE] New Kafka PMC member

答复: [ANNOUNCE] New Kafka PMC member Ismael Juma

2017-07-06 Thread Hu Xi
Congrats, Ismael  发件人: Molnár Bálint 发送时间: 2017年7月6日 15:51 收件人: dev@kafka.apache.org 主题: Re: [ANNOUNCE] New Kafka PMC member Ismael Juma Congrats, Ismael:) 2017-07-06 2:57 GMT+02:00 Matthias J. Sax : > Congrats!

[jira] [Created] (KAFKA-5560) LogManager should be able to create new logs based on free disk space

2017-07-06 Thread huxihx (JIRA)
huxihx created KAFKA-5560: - Summary: LogManager should be able to create new logs based on free disk space Key: KAFKA-5560 URL: https://issues.apache.org/jira/browse/KAFKA-5560 Project: Kafka Issue

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

2017-07-06 Thread Molnár Bálint
Congrats, Ismael:) 2017-07-06 2:57 GMT+02:00 Matthias J. Sax : > Congrats! > > On 7/5/17 3:42 PM, Roger Hoover wrote: > > Well deserved, indeed! Congrats, Ismael. > > > > On Wed, Jul 5, 2017 at 3:24 PM, Damian Guy wrote: > > > >> Congratulations

[GitHub] kafka-site issue #64: MINOR: Add release dates to downloads page

2017-07-06 Thread ijuma
Github user ijuma commented on the issue: https://github.com/apache/kafka-site/pull/64 Good idea. As you said, the dates seem to differ slightly from: https://issues.apache.org/jira/browse/KAFKA/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel It would be

Re: [DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

2017-07-06 Thread Sönke Liebau
I've updated the pull request to behave as follows: - reject create requests that contain no "name" element with a BadRequestException - reject name that are empty or contain illegal characters with a ConfigException - leave current logic around when to copy the name from the create request to