date:20170518

[jira] [Created] (KAFKA-5286) Producer should await transaction completion in close

2017-05-18 Thread Jason Gustafson (JIRA)

Jason Gustafson created KAFKA-5286:
--

 Summary: Producer should await transaction completion in close
 Key: KAFKA-5286
 URL: https://issues.apache.org/jira/browse/KAFKA-5286
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson


We should wait at least as long as the timeout for a transaction which has 
begun completion (commit or abort) to be finished. Tricky thing is whether we 
should abort a transaction which is in progress. It seems reasonable since 
that's the coordinator will either timeout and abort the transaction or the 
next producer using the same transactionalId will fence the producer and abort 
the transaction. In any case, the transaction will be aborted, so perhaps we 
should do it proactively.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk7 #2231

2017-05-18 Thread Apache Jenkins Server

See

Build failed in Jenkins: kafka-trunk-jdk7 #2230

2017-05-18 Thread Apache Jenkins Server

See 


Changes:

[ismael] KAFKA-5250: Do fetch down conversion after throttling

--
[...truncated 819.69 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotOverwriteLogEndOffsetForALeaderEpochOnceItHasBeenAssigned STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotOverwriteLogEndOffsetForALeaderEpochOnceItHasBeenAssigned PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUndefinedOffsetIfUndefinedEpochRequested STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUndefinedOffsetIfUndefinedEpochRequested PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldIncreaseAndTrackEpochsAsLeadersChangeManyTimes STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldIncreaseAndTrackEpochsAsLeadersChangeManyTimes PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldSupportEpochsThatDoNotStartFromZero STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldSupportEpochsThatDoNotStartFromZero PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfRequestedEpochLessThanFirstEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfRequestedEpochLessThanFirstEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnNextAvailableEpochIfThereIsNoExactEpochForTheOneRequested STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnNextAvailableEpochIfThereIsNoExactEpochForTheOneRequested PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldAddEpochAndMessageOffsetToCache STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldAddEpochAndMessageOffsetToCache PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldDropEntriesOnEpochBoundaryWhenRemovingLatestEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldDropEntriesOnEpochBoundaryWhenRemovingLatestEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateSavedOffsetWhenOffsetToClearToIsBetweenEpochs STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateSavedOffsetWhenOffsetToClearToIsBetweenEpochs PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest >

Build failed in Jenkins: kafka-0.11.0-jdk7 #2

2017-05-18 Thread Apache Jenkins Server

See 


Changes:

[wangguoz] KAFKA-5192: add WindowStore range scan (KIP-155)

--
[...truncated 884.98 KB...]
kafka.admin.TopicCommandTest > testCreateIfNotExists PASSED

kafka.admin.TopicCommandTest > testCreateAlterTopicWithRackAware STARTED

kafka.admin.TopicCommandTest > testCreateAlterTopicWithRackAware PASSED

kafka.admin.TopicCommandTest > testTopicDeletion STARTED

kafka.admin.TopicCommandTest > testTopicDeletion PASSED

kafka.admin.TopicCommandTest > testConfigPreservationAcrossPartitionAlteration 
STARTED

kafka.admin.TopicCommandTest > testConfigPreservationAcrossPartitionAlteration 
PASSED

kafka.admin.TopicCommandTest > testAlterIfExists STARTED

kafka.admin.TopicCommandTest > testAlterIfExists PASSED

kafka.admin.TopicCommandTest > testDeleteIfExists STARTED

kafka.admin.TopicCommandTest > testDeleteIfExists PASSED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress STARTED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown PASSED

kafka.controller.ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline STARTED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled PASSED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection STARTED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection PASSED

kafka.controller.ControllerIntegrationTest > testTopicCreation STARTED

kafka.controller.ControllerIntegrationTest > testTopicCreation PASSED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment STARTED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment PASSED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion STARTED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled PASSED

kafka.controller.ControllerIntegrationTest > testEmptyCluster STARTED

kafka.controller.ControllerIntegrationTest > testEmptyCluster PASSED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
STARTED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
PASSED

kafka.controller.ControllerFailoverTest > testMetadataUpdate STARTED

kafka.controller.ControllerFailoverTest > testMetadataUpdate SKIPPED

kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs STARTED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest >

0.11.0.0 Feature Freeze Update

2017-05-18 Thread Ismael Juma

Hi all,

As you hopefully know, are now past the feature freeze for 0.11.0.0. As
such, I have created the 0.11.0 branch, its Jenkins job and bumped the
Kafka version in trunk. From now on, if a change needs to be in 0.11.0.0,
it needs to be merged to trunk and the 0.11.0 branch.

As documented in the release plan, major features should now be merged and
we will be working on stabilization (tests, fixes, documentation, etc.).
Minor features (including minor changes to major features) with a PR can be
merged until next Wednesday (24th May). After that, it's all about fixes,
tests and documentation.

As a reminder, here are the upcoming important dates (also documented in
the release plan https://cwiki.apache.org/confluence/display/KAFKA/Release+
Plan+0.11.0.0):

- Code Freeze: May 31, 2017 (first RC created now)
- Release: June 14, 2017

Not long to go now. :)

KIPs: we have 32 adopted with almost all of them already committed. We have
bumped 7 KIPs to the next release. Now that we have created the 0.11.0
branch, we can start merging PRs for the next release.

Open JIRAs: As usual, we have a lot!

*https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%200.11.0.0%20ORDER%20BY%20priority%20DESC
*

157 at the moment. Now that we are past the feature freeze, I will start
moving JIRAs to the next release.

* Closed JIRAs: So far ~265 closed tickets for 0.11.0.0:
https://issues.apache.org/jira/issues/?jql=project%20%3D%
20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND
%20fixVersion%20%3D%200.11.0.0%20ORDER%20BY%20priority%20DESC

* Release features:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.0 has
a "Release Features" section that will be included with the release
notes/email for the release. I added some items to get it going. Please add
to
this list anything you think is worth noting.

If you missed any of the updates on the new time-based releases we'll be
following, see
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
for an explanation.

I'll plan to give another update a week from now.

Thanks,
Ismael

[jira] [Updated] (KAFKA-5285) optimize upper / lower byte range for key range scan on windowed stores

2017-05-18 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-5285:
-
Labels: performance  (was: )

> optimize upper / lower byte range for key range scan on windowed stores
> ---
>
> Key: KAFKA-5285
> URL: https://issues.apache.org/jira/browse/KAFKA-5285
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
>  Labels: performance
>
> The current implementation of {{WindowKeySchema}} / {{SessionKeySchema}} 
> {{upperRange}} and {{lowerRange}} does not make any assumptions with respect 
> to the other key bound (e.g. the upper byte bound does not depends on lower 
> key bound).
> It should be possible to optimize the byte range somewhat further using the 
> information provided by the lower bound.
> More specifically, by incorporating that information, we should be able to 
> eliminate the corresponding {{upperRangeFixedSize}} and 
> {{lowerRangeFixedSize}}, since the result should be the same if we implement 
> that optimization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5285) optimize upper / lower byte range for key range scan on windowed stores

2017-05-18 Thread JIRA

Xavier Léauté created KAFKA-5285:


 Summary: optimize upper / lower byte range for key range scan on 
windowed stores
 Key: KAFKA-5285
 URL: https://issues.apache.org/jira/browse/KAFKA-5285
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Xavier Léauté
Assignee: Xavier Léauté


The current implementation of {{WindowKeySchema}} / {{SessionKeySchema}} 
{{upperRange}} and {{lowerRange}} does not make any assumptions with respect to 
the other key bound (e.g. the upper byte bound does not depends on lower key 
bound).

It should be possible to optimize the byte range somewhat further using the 
information provided by the lower bound.

More specifically, by incorporating that information, we should be able to 
eliminate the corresponding {{upperRangeFixedSize}} and 
{{lowerRangeFixedSize}}, since the result should be the same if we implement 
that optimization.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

2017-05-18 Thread dan norwood (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016718#comment-16016718
 ] 

dan norwood commented on KAFKA-5275:


it would be nice to have a default {{describeTopics()}} that doesn't need a 
list of topics and just returns the description for all topics, right now i 
have to do 
{{adminClient.describeTopics(a.listTopics().names().get()).all().get();}} :/



> Review and potentially tweak AdminClient API for the initial release (KIP-117)
> --
>
> Key: KAFKA-5275
> URL: https://issues.apache.org/jira/browse/KAFKA-5275
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Once all the pieces are in, we should take a pass and ensure that the APIs 
> work well together and that they are consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3027: KAFKA-5192 WindowStore range scan KIP-155

2017-05-18 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3027


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5192) Range Scan for Windowed State Stores

2017-05-18 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5192.
--
   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 3027
[https://github.com/apache/kafka/pull/3027]

> Range Scan for Windowed State Stores
> 
>
> Key: KAFKA-5192
> URL: https://issues.apache.org/jira/browse/KAFKA-5192
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
> Fix For: 0.11.0.0
>
>
> Windowed state stores currently do not support key range scans, even though 
> it seems reasonable to be able to – at least in a given window – do the same 
> operations you would do on a key-value store.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5192) Range Scan for Windowed State Stores

2017-05-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016708#comment-16016708
 ] 

ASF GitHub Bot commented on KAFKA-5192:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3027


> Range Scan for Windowed State Stores
> 
>
> Key: KAFKA-5192
> URL: https://issues.apache.org/jira/browse/KAFKA-5192
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
> Fix For: 0.11.0.0
>
>
> Windowed state stores currently do not support key range scans, even though 
> it seems reasonable to be able to – at least in a given window – do the same 
> operations you would do on a key-value store.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3095: MINOR: Bump Kafka version to 0.11.1.0-SNAPSHOT

2017-05-18 Thread ijuma

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3095

MINOR: Bump Kafka version to 0.11.1.0-SNAPSHOT



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka bump-kafka-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3095.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3095


commit 8fc273bf379f24212771290dacdfb23f618480d5
Author: Ismael Juma 
Date:   2017-05-18T23:59:54Z

MINOR: Bump Kafka version to 0.11.1.0-SNAPSHOT




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-05-18 Thread Jeyhun Karimov

Hi,

Thanks. I initiated the PR as well, to get a better overview.

The only reason that I used abstract class instead of interface for Rich
functions is that in future if we will have some AbstractRichFunction
abstract classes,
we can easily extend:

public abstract class RichValueMapper  implements
ValueMapperWithKey, RichFunction *extends  AbstractRichFunction*{
}
 With interfaces we are only limited to interfaces for inheritance.

However, I think we can easily change it (from abstract class -> interface)
if you think interface is a better fit.


Cheers,
Jeyhun


On Fri, May 19, 2017 at 1:00 AM Matthias J. Sax 
wrote:

> Thanks for the update and explanations. The KIP is quite improved now --
> great job!
>
> One more question: Why are RichValueMapper etc abstract classes and not
> interfaces?
>
>
> Overall, I like the KIP a lot!
>
>
> -Matthias
>
>
> On 5/16/17 7:03 AM, Jeyhun Karimov wrote:
> > Hi,
> >
> > Thanks for your comments.
> >
> > I think supporting Lambdas for `withKey` and `AbstractRichFunction`
> >> don't go together, as Lambdas are only supported for interfaces AFAIK.
> >
> >
> > Maybe I misunderstood your comment.
> > *withKey* and and *withOnlyValue* are interfaces. So they don't have
> direct
> > relation with *AbstractRichFunction*.
> > *withKey* and and *withOnlyValue* interfaces have only one  method , so
> we
> > can use lambdas.
> > Where does the *AbstractRichFunction* comes to play? Inside Rich
> functions.
> > And we use Rich functions in 2 places:
> >
> > 1. User doesn't use rich functions. Just regular *withKey* and and
> > *withOnlyValue* interfaces(both support lambdas) . We get those
> interfaces
> > and wrap into Rich function while building the topology, and send to
> > Processor.
> > 2. User does use rich functions (Rich functions implement *withKey*
> > interface). As a result no lamdas here by definition. In this case, while
> > building the topology we do a type check if the object type is
> > *withKey* or *RichFunction*.
> >
> > So *AbstractRichFunction* is just syntactic sugar for Rich functions and
> > does not affect using lambdas.
> >
> > Thus, if we want to support Lambdas for `withKey`, we need to have a
> >> interface approach like this
> >>   - RichFunction -> only adding init() and close()
> >>   - ValueMapper
> >>   - ValueMapperWithKey
> >>   - RichValueMapper extends ValueMapperWithKey, RichFunction
> >
> >
> > As I said above, currently we support lambdas for *withKey* interfaces as
> > well.  However, I agree with your idea and I will remove the
> > AbstractRichFunction from the design.
> >
> > As an alternative, we could argue, that it is sufficient to support
> >> Lambdas for the "plain" API only, but not for any "extended API". For
> >> this, RichFunction could add key+init+close and AbstractRichFunction
> >> would allow to only care about getting the key.
> >> Not sure, which one is better. I don't like the idea of more overloaded
> >> methods to get Lambdas for `withKey` interfaces too much because we have
> >> already so many overlaods. On the other hand, I do see value in
> >> supporting Lambdas for `withKey`.
> >
> >
> > Just to clarify, with current design we have only one extra overloaded
> > method per *withOnlyValue* interface:  which is *withKey* version of
> > particular interface.
> > We don't need extra overload for Rich function as Rich function
> implements
> > *withKey* interface as a result they have same type. We differentiate
> them
> > while building the topology.
> > We supported lambdas for *withKey* APIs because of the comment:
> >
> > @Jeyhun: I did not put any thought into this, but can we have a design
> >> that allows for both? Also, with regard to lambdas, it might make sense
> >> to allow for both `V -> newV` and `(K, V) -> newV` ?
> >
> >
> > However, I don't think that this complicates the overall design
> > significantly.
> >
> >
> > Depending on what we want to support, it might make sense to
> >> include/exclude RichFunctions from this KIP -- and thus, this also
> >> determines if we should have a "ProcessorContext KIP" before driving
> >> this KIP further.
> >
> >
> > Based on our discussion I think we should keep Rich functions as I don't
> > think that they bring extra layer of overhead to library.
> >
> > Any comments are appreciated.
> >
> > Cheers,
> > Jeyhun
> >
> >
> > On Tue, May 16, 2017 at 12:10 AM Matthias J. Sax 
> > wrote:
> >
> >> Jeyhun,
> >>
> >> thanks for the update.
> >>
> >> I think supporting Lambdas for `withKey` and `AbstractRichFunction`
> >> don't go together, as Lambdas are only supported for interfaces AFAIK.
> >>
> >> Thus, if we want to support Lambdas for `withKey`, we need to have a
> >> interface approach like this
> >>
> >>   - RichFunction -> only adding init() and close()
> >>
> >>   - ValueMapper
> >>   - ValueMapperWithKey
> >>
> >>   - RichValueMapper extends ValueMapperWithKey, RichFunction
> >>
> >> For this

[jira] [Comment Edited] (KAFKA-877) Still getting kafka.common.NotLeaderForPartitionException

2017-05-18 Thread Robert P. Thille (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016655#comment-16016655
 ] 

Robert P. Thille edited comment on KAFKA-877 at 5/18/17 11:32 PM:
--

I have logs from a 3-node cluster which got into a bad state after ZK expired:
{noformat}
[2017-05-09 18:33:42,897] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 05:06:13,469] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 16:33:43,349] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 16:33:44,059] INFO [Controller 2]: Broker 2 starting become 
controller state transition (kafka.controller.KafkaController)
{noformat}
The first two ZK session expirations were handled fine. The 3rd blew everything 
up.


was (Author: rthille):
I have logs from a 3-node cluster which got into a bad state after ZK expired:
{noformat}
[2017-05-09 18:33:42,897] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 05:06:13,469] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 16:33:43,349] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 16:33:44,059] INFO [Controller 2]: Broker 2 starting become 
controller state transition (kafka.controller.KafkaController)
{noformat}

> Still getting kafka.common.NotLeaderForPartitionException
> -
>
> Key: KAFKA-877
> URL: https://issues.apache.org/jira/browse/KAFKA-877
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
> Environment: DEV
>Reporter: BalajiSeshadri
>Priority: Blocker
> Attachments: KAFKA-816.jpg
>
>
> Using the below trunk and i still see error happening.Please let us know if 
> this can be fixed.
> https://github.com/apache/kafka.git
> [2013-04-25 16:47:08,924] WARN 
> [console-consumer-24019_MERD7-21964-1366930009136-8b7f9eb7-leader-finder-thread],
>  Failed to add fetcher for [mytopic,0] to broker 
> id:0,host:MERD7-21964.echostar.com,port:9092 
> (kafka.consumer.ConsumerFetcherManager$$anon$1)
> kafka.common.NotLeaderForPartitionException
> at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown 
> Source)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at java.lang.Class.newInstance0(Class.java:372)
> at java.lang.Class.newInstance(Class.java:325)
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:72)
> at 
> kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:163)
> at 
> kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:61)
> at 
> kafka.server.AbstractFetcherThread.addPartition(AbstractFetcherThread.scala:167)
> at 
> kafka.server.AbstractFetcherManager.addFetcher(AbstractFetcherManager.scala:48)
> at 
> kafka.consumer.ConsumerFetcherManager$$anon$1$$anonfun$doWork$3.apply(ConsumerFetcherManager.scala:79)
> at 
> kafka.consumer.ConsumerFetcherManager$$anon$1$$anonfun$doWork$3.apply(ConsumerFetcherManager.scala:75)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> at 
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
> at 
> kafka.consumer.ConsumerFetcherManager$$anon$1.doWork(ConsumerFetcherManager.scala:75)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> We are evaluating Kafka for our new messaging system and we had tough time 
> running in windows.
> We somehow managed to run 0.8 using cygwin but when we run the console 
>

[jira] [Commented] (KAFKA-877) Still getting kafka.common.NotLeaderForPartitionException

2017-05-18 Thread Robert P. Thille (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016655#comment-16016655
 ] 

Robert P. Thille commented on KAFKA-877:


I have logs from a 3-node cluster which got into a bad state after ZK expired:
{noformat}
[2017-05-09 18:33:42,897] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 05:06:13,469] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 16:33:43,349] INFO [SessionExpirationListener on 2], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-05-17 16:33:44,059] INFO [Controller 2]: Broker 2 starting become 
controller state transition (kafka.controller.KafkaController)
{noformat}

> Still getting kafka.common.NotLeaderForPartitionException
> -
>
> Key: KAFKA-877
> URL: https://issues.apache.org/jira/browse/KAFKA-877
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
> Environment: DEV
>Reporter: BalajiSeshadri
>Priority: Blocker
> Attachments: KAFKA-816.jpg
>
>
> Using the below trunk and i still see error happening.Please let us know if 
> this can be fixed.
> https://github.com/apache/kafka.git
> [2013-04-25 16:47:08,924] WARN 
> [console-consumer-24019_MERD7-21964-1366930009136-8b7f9eb7-leader-finder-thread],
>  Failed to add fetcher for [mytopic,0] to broker 
> id:0,host:MERD7-21964.echostar.com,port:9092 
> (kafka.consumer.ConsumerFetcherManager$$anon$1)
> kafka.common.NotLeaderForPartitionException
> at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown 
> Source)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at java.lang.Class.newInstance0(Class.java:372)
> at java.lang.Class.newInstance(Class.java:325)
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:72)
> at 
> kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:163)
> at 
> kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:61)
> at 
> kafka.server.AbstractFetcherThread.addPartition(AbstractFetcherThread.scala:167)
> at 
> kafka.server.AbstractFetcherManager.addFetcher(AbstractFetcherManager.scala:48)
> at 
> kafka.consumer.ConsumerFetcherManager$$anon$1$$anonfun$doWork$3.apply(ConsumerFetcherManager.scala:79)
> at 
> kafka.consumer.ConsumerFetcherManager$$anon$1$$anonfun$doWork$3.apply(ConsumerFetcherManager.scala:75)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> at 
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
> at 
> kafka.consumer.ConsumerFetcherManager$$anon$1.doWork(ConsumerFetcherManager.scala:75)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> We are evaluating Kafka for our new messaging system and we had tough time 
> running in windows.
> We somehow managed to run 0.8 using cygwin but when we run the console 
> producer/consumer,we are not getting messages from consumer.
> Please help us to fix this issue,this might not be related but its keeping on 
> throwing this error on consumer side. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

1 2 >

1 - 100 of 167 matches

Mail list logo