[jira] [Resolved] (KAFKA-5141) WorkerTest.testCleanupTasksOnStop transient failure due to NPE

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-5141.
--
Resolution: Fixed
  Assignee: Ewen Cheslack-Postava

Not sure of the fix, but we haven't seen this test failure in a long time. 
Pretty sure it has been fixed somewhere along the way. We can re-open if we see 
the same issue again.

> WorkerTest.testCleanupTasksOnStop transient failure due to NPE
> --
>
> Key: KAFKA-5141
> URL: https://issues.apache.org/jira/browse/KAFKA-5141
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Major
>  Labels: transient-unit-test-failure
>
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/3281/testReport/junit/org.apache.kafka.connect.runtime/WorkerTest/testCleanupTasksOnStop/
> Looks like the potential culprit is a NullPointerException when trying to 
> start a connector. It's likely being caught and logged via a catch 
> (Throwable). From the lines being executed it looks like the null might be 
> due to the instantiation of the Connector returning null, although I don't 
> see how that is possible given the current code. We may need more logging 
> output to track the issue down.
> {quote}
> Error Message
> java.lang.AssertionError: 
>   Expectation failure on verify:
> WorkerSourceTask.run(): expected: 1, actual: 0
> Stacktrace
> java.lang.AssertionError: 
>   Expectation failure on verify:
> WorkerSourceTask.run(): expected: 1, actual: 0
>   at org.easymock.internal.MocksControl.verify(MocksControl.java:225)
>   at 
> org.powermock.api.easymock.internal.invocationcontrol.EasyMockMethodInvocationControl.verify(EasyMockMethodInvocationControl.java:132)
>   at org.powermock.api.easymock.PowerMock.verify(PowerMock.java:1466)
>   at org.powermock.api.easymock.PowerMock.verifyAll(PowerMock.java:1405)
>   at 
> org.apache.kafka.connect.runtime.WorkerTest.testCleanupTasksOnStop(WorkerTest.java:480)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:310)
>   at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
>   at 
> org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:294)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:127)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:282)
>   at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
>   at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:50)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:207)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:146)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:120)
>   at 
> org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
>   at 
> org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
>   at 
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:122)
>   at 
> org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:106)
>   at 
> org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:53)
>   at 
> 

Re: KIP 244 WIP

[Ted Yu]

The KIP is still under discussion.

Have you seen this thread ?

http://search-hadoop.com/m/Kafka/uyzND1cGBCC19brXW?subj=Re+KIP+244+Add+Record+Header+support+to+Kafka+Streams

FYI

On Sun, Apr 29, 2018 at 10:29 AM, Florian Garcia <
garcia.florian.pe...@gmail.com> wrote:

> Hello,
>
> I am really interested about the "KIP-244
>  : Add record
> headers to Kafka Streams".
> I saw that the previous implementation hasn't change since december so I
> decided to start my own during this weekend. For now, only the
> `.filter((key, value, header) -> ...)` is working (Tests are in
> KStreamFilterTest.java). You can find the changes here:
> https://github.com/apache/kafka/compare/trunk...ImFlog:streams-headers
>
> Can someone take a quick look at this and tell me if I'm heading toward the
> right direction ?
> Some things feels quite wrong as I am duplicating a lot because of the
> AbstractProcessor new `process` methods and the `forward` methods
> from ProcessorContextImpl.java.
>
> If the filter is well implemented, I will continue with the rest.
>
> Thank you by advance
> Florian Garcia
>

Fwd: KIP 244 WIP

[Florian Garcia]

Hello,

I am really interested about the "KIP-244
 : Add record
headers to Kafka Streams".
I saw that the previous implementation hasn't change since december so I
decided to start my own during this weekend. For now, only the
`.filter((key, value, header) -> ...)` is working (Tests are in
KStreamFilterTest.java). You can find the changes here:
https://github.com/apache/kafka/compare/trunk...ImFlog:streams-headers

Can someone take a quick look at this and tell me if I'm heading toward the
right direction ?
Some things feels quite wrong as I am duplicating a lot because of the
AbstractProcessor new `process` methods and the `forward` methods
from ProcessorContextImpl.java.

If the filter is well implemented, I will continue with the rest.

Thank you by advance
Florian Garcia

[jira] [Created] (KAFKA-6836) Upgrade metrics library

[Mario Molina (JIRA)]

Mario Molina created KAFKA-6836:
---

 Summary: Upgrade metrics library
 Key: KAFKA-6836
 URL: https://issues.apache.org/jira/browse/KAFKA-6836
 Project: Kafka
  Issue Type: Improvement
  Components: metrics
Affects Versions: 1.1.0
Reporter: Mario Molina
Assignee: Mario Molina
 Fix For: 2.0.0, 1.1.1


The current metrics library which Kafka is using is pretty old (version 2.2.0 
from Yammer and now we have 4.X from Dropwizard).

In the latest versions of the Dropwizard library (which comes from Yammer and 
this is deprecated), there are a lot of bugfixes and new features included 
which could be interesting for the metrics (ie: Reservoris, support JDK9, etc).

This patch includes the upgrade to this new version of the library so that we 
could add new features in Kafka metrics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2604

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] MINOR: Fix Trogdor tests, partition assignments (#4892)

[rajinisivaram] KAFKA-6785: Add Trogdor documentation (#4862)

--
[...truncated 419.10 KB...]
kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLatest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLatest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption PASSED

[jira] [Created] (KAFKA-6835) Enable topic unclean leader election to be enabled without controller change

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6835:
-

 Summary: Enable topic unclean leader election to be enabled 
without controller change
 Key: KAFKA-6835
 URL: https://issues.apache.org/jira/browse/KAFKA-6835
 Project: Kafka
  Issue Type: Task
  Components: core
Reporter: Rajini Sivaram


Dynamic update of broker's default unclean.leader.election.enable will be 
processed without controller change (KAFKA-6526). We should probably do the 
same for topic overrides as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-235 Add DNS alias support for secured connection

[Rajini Sivaram]

Hi Jonathan,

Thanks for the KIP.

+1 (binding) with a couple comments below to add more detail to the KIP.

   1. Make it clearer when the new option `bootstrap.reverse.dns.lookup`
   should or shouldn't be used. Document security considerations as well as
   other system configurations that may have an impact.
   2. The PR currently disables the new code path for security protocol
   SSL. But this doesn't address SASL_SSL which could also do hostname
   verification. Do we even want to do reverse lookup if bootstrap list
   contains IP addresses? If we do, we should handle SSL and SASL_SSL in the
   same way (which basically means handling all protocols in the same way).


On Thu, Apr 26, 2018 at 2:16 PM, Stephane Maarek <
steph...@simplemachines.com.au> wrote:

> +1 as a user
> BUT
>
> I am no security expert. I have experienced that issue while setting up a
> cluster and while I would have liked a feature like that (I opened a JIRA
> at the time), I always guessed that the reason was because of some security
> protection.
>
> Now from a setup point of view this helps a ton, but I really want to make
> sure this doesn't introduce any security risk by relaxing a constraint.
>
> Is there a security assessment possible by someone accredited ?
>
> Sorry for raising these questions just want to make sure it's addressed
>
> On Thu., 26 Apr. 2018, 5:32 pm Gwen Shapira,  wrote:
>
> > +1 (binding)
> >
> > This KIP is quite vital to running secured clusters in cloud/container
> > environment. Would love to see more support from the community to this
> (or
> > feedback...)
> >
> > Gwen
> >
> > On Mon, Apr 16, 2018 at 4:52 PM, Skrzypek, Jonathan <
> > jonathan.skrzy...@gs.com> wrote:
> >
> > > Hi,
> > >
> > > Could anyone take a look ?
> > > Does the proposal sound reasonable ?
> > >
> > > Jonathan Skrzypek
> > >
> > >
> > > From: Skrzypek, Jonathan [Tech]
> > > Sent: 23 March 2018 19:05
> > > To: dev@kafka.apache.org
> > > Subject: [VOTE] KIP-235 Add DNS alias support for secured connection
> > >
> > > Hi,
> > >
> > > I would like to start a vote for KIP-235
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 235%3A+Add+DNS+alias+support+for+secured+connection
> > >
> > > This is a proposition to add an option for reverse dns lookup of
> > > bootstrap.servers hosts, allowing the use of dns aliases on clusters
> > using
> > > SASL authentication.
> > >
> > >
> > >
> > >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter  | blog
> > 
> >
>