join kafka

[Xu Jianhai]

join kafka

Jenkins build is back to normal : kafka-trunk-jdk11 #840

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-8958) Fix Kafka Streams JavaDocs with regard to used Serdes

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-8958:
--

 Summary: Fix Kafka Streams JavaDocs with regard to used Serdes
 Key: KAFKA-8958
 URL: https://issues.apache.org/jira/browse/KAFKA-8958
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


In older released, Kafka Streams applied operator specific overwrites of Serdes 
as in-place overwrites. In newer releases, Kafka Streams tries to re-use Serdes 
more "aggressively" by pushing serde information downstream if the key and/or 
value did not change.

However, we never updated the JavaDocs accordingly. For example 
`KStream#through(String topic)` JavaDocs say:
{code:java}
Materialize this stream to a topic and creates a new {@code KStream} from the 
topic using default serializers, deserializers, and producer's {@link 
DefaultPartitioner}.
{code}
The JavaDocs don't put into account that Serdes might have been set further 
upstream, and the defaults from the config would not be used.

`KStream#through()` is just one example. We should address this through all 
JavaDocs over all operators (ie, KStream, KGroupedStream, TimeWindowedKStream, 
SessionWindowedKStream, KTable, and KGroupedTable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #3928

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-8907; Return topic configs in CreateTopics response (KIP-525)

--
[...truncated 6.86 MB...]
org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldUseRecordMetadataTimestampExtractorWhenInternalRepartitioningTopicCreatedWithRetention
 STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldUseRecordMetadataTimestampExtractorWhenInternalRepartitioningTopicCreatedWithRetention
 PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > testNumProcesses 
STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > testNumProcesses 
PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullValueTransformerWithKeySupplierOnTransformValues STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullValueTransformerWithKeySupplierOnTransformValues PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldThrowNullPointerOnJoinWithTableWhenJoinedIsNull STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldThrowNullPointerOnJoinWithTableWhenJoinedIsNull PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnFlatMapValuesWithKey STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnFlatMapValuesWithKey PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldProcessFromSourcesThatMatchMultiplePattern STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldProcessFromSourcesThatMatchMultiplePattern PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnFlatMapValues STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnFlatMapValues PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullValueTransformerSupplierOnTransformValues STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullValueTransformerSupplierOnTransformValues PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnSelectKey STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnSelectKey PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldMergeMultipleStreams STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldMergeMultipleStreams PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullTopicOnTo STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullTopicOnTo PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullValueTransformerSupplierOnFlatTransformValues STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullValueTransformerSupplierOnFlatTransformValues PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnMapValuesWithKey STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnMapValuesWithKey PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldUseRecordMetadataTimestampExtractorWhenInternalRepartitioningTopicCreated 
STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldUseRecordMetadataTimestampExtractorWhenInternalRepartitioningTopicCreated 
PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnMap STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullMapperOnMap PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullJoinWindowsOnJoin STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullJoinWindowsOnJoin PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullJoinerOnLeftJoinWithGlobalTable STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullJoinerOnLeftJoinWithGlobalTable PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullJoinerOnJoinWithGlobalTable STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullJoinerOnJoinWithGlobalTable PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullValueJoinerOnJoin STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullValueJoinerOnJoin PASSED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullTableOnJoinWithGlobalTable STARTED

org.apache.kafka.streams.kstream.internals.KStreamImplTest > 
shouldNotAllowNullTableOnJoinWithGlobalTable PASSED

Build failed in Jenkins: kafka-trunk-jdk11 #839

[Apache Jenkins Server]

See 


Changes:

[manikumar] KAFKA-6883: Add toUpperCase support to 
sasl.kerberos.principal.to.local

[github] KAFKA-8319: Make KafkaStreamsTest a non-integration test class (#7382)

[wangguoz] KAFKA-8934: Create version file during build for Streams (#7397)

--
[...truncated 2.64 MB...]
org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

> Task :streams:streams-scala:test

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes PASSED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped should 
create a Grouped with Serdes STARTED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped should 
create a Grouped with Serdes PASSED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped with 
repartition topic name should create a Grouped with Serdes, and repartition 
topic name STARTED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped with 
repartition topic name should create a Grouped with Serdes, and repartition 
topic name PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using 

Re: [DISCUSS] KIP-515: Reorganize checkpoint system in log cleaner to be per partition

[Richard Yu]

Hi Jason,

That actually sounds like a pretty good idea to me. No doubt if we use this
approach, then some comments need to be added that indicates this.
But all things considered, I think its not bad at all.

I definitely agree with you on that its a little hacky, but it works.

Cheers,
Richard

On Tue, Sep 24, 2019 at 10:44 AM Jason Gustafson  wrote:

> Hi Richard,
>
> It would be unsatisfying to make a big change to the checkpointing logic in
> order to handle only one case of this problem, right?
>
> I did have one idea about how to do this. It's a bit of a hack, but keep an
> open mind ;). The basic problem is having somewhere to embed the delete
> horizon for each batch. In the v2 format, each batch header contains two
> timestamps: the base timestamp and the max timestamp. Each record in the
> batch contains a timestamp delta which is relative to the base timestamp.
> In other words, to get the record timestamp, you add the record delta to
> the base timestamp.
>
> Typically there is no reason for the base timestamp to be different from
> the timestamp of the first message, but this is not a strict requirement.
> As long as you can get to the record timestamp by adding the base timestamp
> and delta, then we are good. So the idea is to set the base timestamp to
> the delete horizon and adjust the deltas accordingly. We could then use one
> bit from the batch attributes to indicate when the base timestamp had been
> set to the delete horizon. There would be no change to the batch max
> timestamp, so indexing would not be affected by this change.
>
> So the logic would look something like this when cleaning the log.
>
> Case 1: Normal batch
>
> a. If delete horizon flag is set, then retain tombstones as long as the
> current time is before the horizon.
> b. If no delete horizon is set, then retain tombstones and set the delete
> horizon in the cleaned batch to current time +
> log.cleaner.delete.retention.ms.
>
> Case 2: Control batch
>
> a. If delete horizon flag is set, then retain the batch and the marker
> as long as the current time is before the horizon.
> b. If no delete horizon is set and there are no records remaining from the
> transaction, then retain the marker and set the delete horizon in the
> cleaned batch to current time + log.cleaner.delete.retention.ms.
>
> What do you think?
>
> -Jason
>
>
>
> On Thu, Sep 19, 2019 at 3:21 PM Richard Yu 
> wrote:
>
> > Hi Jason,
> >
> > That hadn't occurred to me.
> >
> > I think I missed your comment in the discussion, so I created this KIP
> only
> > with resolving the problem regarding tombstones.
> > Whats your thoughts? If the problem regarding transaction markers is a
> > little too complex, then we can we just leave it out of the KIP and fix
> the
> > tombstones issue.
> >
> > Cheers,
> > Richard
> >
> > On Thu, Sep 19, 2019 at 8:47 AM Jason Gustafson 
> > wrote:
> >
> > > Hi Richard,
> > >
> > > Just reposting my comment from the JIRA:
> > >
> > > The underlying problem here also impacts the cleaning of transaction
> > > markers. We use the same delete horizon in order to tell when it is
> safe
> > to
> > > remove the marker. If all the data from a transaction has been cleaned
> > and
> > > the delete horizon has passed enough time, then the marker is eligible
> > for
> > > deletion.
> > >
> > > However, I don't think the same approach that we're proposing to fix
> the
> > > problem for tombstones will work transaction markers. What we need to
> > track
> > > is the timestamp when all the records from a transaction have been
> > removed.
> > > That is when we start the timer for deletion. But this would be
> different
> > > for every transaction and there is no guarantee that earlier
> transactions
> > > will be eligible for deletion before later ones. It all depends on the
> > keys
> > > written in the transaction. I don't see an obvious way to solve this
> > > problem without some record-level bookkeeping, but I might be missing
> > > something.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Mon, Sep 9, 2019 at 7:21 PM Richard Yu 
> > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for chipping in. :)
> > > >
> > > > The description you provided is pretty apt in describing the
> motivation
> > > of
> > > > the KIP, so I will add it. I've made some changes to the KIP and
> > outlined
> > > > the basic approaches of what we have so far (basically changing the
> > > > checkpoint file organization or incorporating an extra internal
> header
> > > > field for a record). I will expand on them shortly.
> > > >
> > > > Any comments are appreciated!
> > > >
> > > > Cheers,
> > > > Richard
> > > >
> > > > On Mon, Sep 9, 2019 at 3:10 PM Jun Rao  wrote:
> > > >
> > > > > Hi, Richard,
> > > > >
> > > > > Thanks for drafting the KIP. A few comments below.
> > > > >
> > > > > 1. We need to provide a better motivation for the KIP. The goal of
> > the
> > > > KIP
> > > > > is not to reorganize the checkpoint for log cleaning. It's just an
> > > > > 

Build failed in Jenkins: kafka-2.1-jdk8 #231

[Apache Jenkins Server]

See 


Changes:

[matthias] KAFKA-7895: Revert suppress changelog bugfix for 2.1 (#7373)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H31 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/2.1^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.1^{commit} # timeout=10
Checking out Revision d90c5ca32e3582cb13a08671f9382e050b22fabd 
(refs/remotes/origin/2.1)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d90c5ca32e3582cb13a08671f9382e050b22fabd
Commit message: "KAFKA-7895: Revert suppress changelog bugfix for 2.1 (#7373)"
 > git rev-list --no-walk 2f83b483e1045cb0b7c91f63e5313555e6c968fb # timeout=10
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
[kafka-2.1-jdk8] $ /bin/bash -xe /tmp/jenkins913385396846173821.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.8.1/bin/gradle
/tmp/jenkins913385396846173821.sh: line 4: 
/home/jenkins/tools/gradle/4.8.1/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] No files found. Configuration error?
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
No credentials specified
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=d90c5ca32e3582cb13a08671f9382e050b22fabd, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #230
Recording test results
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1

Jenkins build is back to normal : kafka-trunk-jdk11 #838

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-8427) Error while cleanup under windows for EmbeddedKafkaCluster

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-8427.
--
Fix Version/s: 2.4.0
 Assignee: Guozhang Wang
   Resolution: Fixed

Should have been fixed via https://github.com/apache/kafka/pull/7382

> Error while cleanup under windows for EmbeddedKafkaCluster
> --
>
> Key: KAFKA-8427
> URL: https://issues.apache.org/jira/browse/KAFKA-8427
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.0.0, 2.2.0
>Reporter: Sukumaar Mane
>Assignee: Guozhang Wang
>Priority: Major
>  Labels: kafka, testing, win10, windows
> Fix For: 2.4.0
>
>
> Unable to run a simple test case for EmbeddedKafkaCluster where there is an 
> object of  EmbeddedKafkaCluster with 1 broker.
>  Running below simple code (which is actually code snippet from 
> *org.apache.kafka.streams.KafkaStreamsTest* class)
> {code:java}
> public class KTest {
> private static final int NUM_BROKERS = 1;
> // We need this to avoid the KafkaConsumer hanging on poll
> // (this may occur if the test doesn't complete quickly enough)
> @ClassRule
> public static final EmbeddedKafkaCluster CLUSTER = new 
> EmbeddedKafkaCluster(NUM_BROKERS);
> private static final int NUM_THREADS = 2;
> private final StreamsBuilder builder = new StreamsBuilder();
> @Rule
> public TestName testName = new TestName();
> private KafkaStreams globalStreams;
> private Properties props;
> @Before
> public void before() {
> props = new Properties();
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, "appId");
> props.put(StreamsConfig.CLIENT_ID_CONFIG, "clientId");
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, 
> CLUSTER.bootstrapServers());
> props.put(StreamsConfig.METRIC_REPORTER_CLASSES_CONFIG, 
> MockMetricsReporter.class.getName());
> props.put(StreamsConfig.STATE_DIR_CONFIG, 
> TestUtils.tempDirectory().getPath());
> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, NUM_THREADS);
> globalStreams = new KafkaStreams(builder.build(), props);
> }
> @After
> public void cleanup() {
> if (globalStreams != null) {
> globalStreams.close();
> }
> }
> @Test
> public void thisIsFirstFakeTest() {
> assert true;
> }
> }
> {code}
> But getting these error message at the time of cleanup
> {code:java}
> java.nio.file.FileSystemException: 
> C:\Users\Sukumaar\AppData\Local\Temp\kafka-3445189010908127083\version-2\log.1:
>  The process cannot access the file because it is being used by another 
> process.
>   at 
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
>   at 
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
>   at 
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
>   at 
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at org.apache.kafka.common.utils.Utils$2.visitFile(Utils.java:753)
>   at org.apache.kafka.common.utils.Utils$2.visitFile(Utils.java:742)
>   at java.nio.file.Files.walkFileTree(Files.java:2670)
>   at java.nio.file.Files.walkFileTree(Files.java:2742)
>   at org.apache.kafka.common.utils.Utils.delete(Utils.java:742)
>   at kafka.zk.EmbeddedZookeeper.shutdown(EmbeddedZookeeper.scala:65)
>   at 
> org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.stop(EmbeddedKafkaCluster.java:122)
>   at 
> org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.after(EmbeddedKafkaCluster.java:151)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
> One similar issue (KAFKA-6075) had been reported and marked as resolved but 
> still getting the error while cleanup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-8319) Flaky Test KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-8319.
--
Fix Version/s: 2.4.0
 Assignee: Guozhang Wang  (was: Bill Bejeck)
   Resolution: Fixed

Should have been fixed via https://github.com/apache/kafka/pull/7382

> Flaky Test KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory
> --
>
> Key: KAFKA-8319
> URL: https://issues.apache.org/jira/browse/KAFKA-8319
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Bill Bejeck
>Assignee: Guozhang Wang
>Priority: Major
>  Labels: flaky-test
> Fix For: 2.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-8957) Improve docs about `min.isr.` and `acks=all`

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-8957:
--

 Summary: Improve docs about `min.isr.` and `acks=all`
 Key: KAFKA-8957
 URL: https://issues.apache.org/jira/browse/KAFKA-8957
 Project: Kafka
  Issue Type: Improvement
  Components: clients, core
Reporter: Matthias J. Sax


The current docs are as follows:
{code:java}
acks=all

This means the leader will wait for the full set of in-sync replicas to 
acknowledge the record. This guarantees that the record will not be lost as 
long as at least one in-sync replica remains alive. This is the strongest 
available guarantee.{code}
{code:java}
min.in.sync.replicas
When a producer sets acks to "all" (or -1), this configuration specifies the 
minimum number of replicas that must acknowledge a write for the write to be 
considered successful. If this minimum cannot be met, then the producer will 
raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend). 
When used together, `min.insync.replicas` and `acks` allow you to enforce 
greater durability guarantees. A typical scenario would be to create a topic 
with a replication factor of 3, set min.insync.replicas to 2, and produce with 
acks of "all". This will ensure that the producer raises an exception if a 
majority of replicas do not receive a write.
{code}
The miss leading part seems to be:

 
{noformat}
the minimum number of replicas that must acknowledge the write
{noformat}
That could be interpreted to mean that the producer request can return 
*_before_* all replicas acknowledge the write. However, min.irs is a 
configuration that aims to specify how many replicase must be online, to 
consider a partition to be available.

The actual behavior is the following (with replication factor = 3 and min.isr = 
2)
 * If all three replicas are in-sync, brokers only ack to the producer after 
all three replicas got the data. (ie, both follows need to ack)
 * However, if one replicas lags (is not in-sync any longer), we are also ok to 
ack to the producer after the remaining in-sync follower acked.

It's *_not_* the case, that if all three replicase are in-sync, brokers ack to 
the producer after one follower acked to the leader.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-6883) KafkaShortnamer should allow to convert Kerberos principal name to upper case user name

[Manikumar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6883.
--
Fix Version/s: 2.4.0
   Resolution: Fixed

Issue resolved by pull request 7375
[https://github.com/apache/kafka/pull/7375]

> KafkaShortnamer should allow to convert Kerberos principal name to upper case 
> user name
> ---
>
> Key: KAFKA-6883
> URL: https://issues.apache.org/jira/browse/KAFKA-6883
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Attila Sasvári
>Assignee: Manikumar
>Priority: Major
> Fix For: 2.4.0
>
>
> KAFKA-5764 implemented support to convert Kerberos principal name to lower 
> case Linux user name via auth_to_local rules. 
> As a follow-up, KafkaShortnamer could be further extended to allow converting 
> principal names to uppercase by appending /U to the rule.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-8956) Refactor DelayedCreatePartitions#updateWaiting to avoid modifying collection in foreach

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-8956:
---

 Summary: Refactor DelayedCreatePartitions#updateWaiting to avoid 
modifying collection in foreach
 Key: KAFKA-8956
 URL: https://issues.apache.org/jira/browse/KAFKA-8956
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe
Assignee: Colin McCabe


We should refactor {{DelayedCreatePartitions#updateWaiting}} to avoid modifying 
the {{waitingPartitions}} collection in its own foreach.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-8955) Add an AbstractResponse#errorCounts method that takes a stream or iterable

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-8955:
---

 Summary: Add an AbstractResponse#errorCounts method that takes a 
stream or iterable
 Key: KAFKA-8955
 URL: https://issues.apache.org/jira/browse/KAFKA-8955
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe


We should have an AbstractResponse#errorCounts method that takes a stream or 
iterable.  This would allow us to avoid copying data in many cases.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Discuss] - KIP-532 - Add KStream#toTable to the Streams DSL

[John Roesler]

Looks good to me! I have no further comments.

Thanks again for the KIP, Aishwarya!
-John

On Fri, Sep 27, 2019 at 10:11 AM aishwarya kumar  wrote:
>
> Hello John,
>
> Thank you for pointing this out to me, to maintain consistency across API's
> it does make sense to allow users to define custom names for
> their processors.
>
> I've made the change in the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-523%3A+Add+KStream%23toTable+to+the+Streams+DSL
>
> Best,
> Aishwarya
>
> On Tue, Sep 24, 2019 at 11:54 AM John Roesler  wrote:
>
> > Hey Aishwarya,
> >
> > Thanks for the KIP! It looks good to me, although in a post-KIP-307
> > world, we also need a "Named" parameter (to give the processor node a
> > name, as opposed to the store itself).
> >
> > This would result in a total of four overloads:
> > 1. no args
> > 2. Named
> > 3. Materialized
> > 4. Materialized, Named
> >
> > I'd like to propose a re-design of the DSL in the future to clean this
> > up, but for now, this is the pattern we have to follow.
> >
> > Thoughts?
> >
> > Thanks,
> > -John
> >
> > On Tue, Sep 24, 2019 at 9:54 AM aishwarya kumar 
> > wrote:
> > >
> > > Thank you for the suggestion Matthais, i've made the necessary changes in
> > > the KIP.
> > >
> > > Keeping this thread open for further input.
> > > KIP link:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-523%3A+Add+KStream%23toTable+to+the+Streams+DSL
> > >
> > > Best,
> > > Aishwarya
> > >
> > > On Thu, Sep 19, 2019 at 10:50 AM aishwarya kumar 
> > wrote:
> > >
> > > > Thanks Matthias,
> > > >
> > > > That does make sense, let me update the KIP to reflect the
> > Materialization
> > > > scenario.
> > > >
> > > > Best,
> > > > Aishwarya
> > > >
> > > > On Tue, Sep 17, 2019, 2:49 PM Matthias J. Sax 
> > > > wrote:
> > > >
> > > >> Aishwarya,
> > > >>
> > > >> thanks for the KIP. Overall, I think it makes sense to allow
> > converting
> > > >> a KStream into a KTable.
> > > >>
> > > >> From the KIP:
> > > >>
> > > >> > materializing these KTables should only be allowed if the overloaded
> > > >> function with Materialized is used (and if optimization is turned on
> > it may
> > > >> still be only logically materialized if the queryable name is not
> > set).
> > > >>
> > > >> Can you elaborate? I think the behavior we want should align with the
> > > >> behavior of `StreamsBuilder#table()`.
> > > >>
> > > >> From my understanding (correct me if I am wrong) it should be:
> > > >>
> > > >> (1) If optimization is turned off, the KTable will always be
> > > >> materialized, independent which method is used. The KTable will not be
> > > >> queryable though.
> > > >>
> > > >> (2) If optimization is turned on and if `toTable()` is used, the
> > KTable
> > > >> may or may not be materialized. For this case, even if the KTable is
> > > >> materialized, the store would not be queryable.
> > > >>
> > > >> (3) If `toTable(Materialized)` is use and a `storeName` or
> > > >> `StoreSupplier` is specified, the store will always be materialized
> > and
> > > >> also be queryable. Otherwise, case (1) or (2) applies.
> > > >>
> > > >>
> > > >>
> > > >> -Matthias
> > > >>
> > > >>
> > > >> On 9/17/19 6:42 AM, aishwarya kumar wrote:
> > > >> > Hi All,
> > > >> >
> > > >> > Keeping this thread alive!!
> > > >> >
> > > >> > The aim is to add two methods Kstream.toTable() &
> > > >> > Kstream.toTable(Materialized), so users can choose to convert
> > their
> > > >> > event stream into a changelog stream at any stage.
> > > >> > wiki link :
> > > >> >
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-523:+Add+KStream%23toTable+to+the+Streams+DSL
> > > >> > jira ticket : https://issues.apache.org/jira/browse/KAFKA-7658
> > > >> >
> > > >> > Best,
> > > >> > Aishwarya
> > > >> >
> > > >> > On Fri, Sep 13, 2019 at 10:49 AM aishwarya kumar <
> > ash26...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> >> Hello,
> > > >> >>
> > > >> >> Starting this thread to discuss KIP-532:
> > > >> >> wiki link :
> > > >> >>
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-523:+Add+KStream%23toTable+to+the+Streams+DSL
> > > >> >> jira ticket : https://issues.apache.org/jira/browse/KAFKA-7658
> > > >> >>
> > > >> >> There has been some discussion around the use-case of this KIP in
> > the
> > > >> Jira
> > > >> >> ticket.
> > > >> >>
> > > >> >> Regards,
> > > >> >> Aishwarya
> > > >> >>
> > > >> >
> > > >>
> > > >>
> >

Re: [DISCUSS] KIP-424: Allow suppression of intermediate events based on wall clock time

[John Roesler]

Hey Jonathan,

What's the status of this KIP? I was just in a discussion about
suppress, and it sparked my memory of this idea.

Thanks,
-John

On Mon, Mar 11, 2019 at 4:39 PM John Roesler  wrote:
>
> Thanks for the response, Matthias, I agree on both of these points.
>
> I didn't mean to question whether we should discuss it; we should, since as 
> you point out both points affect the behavior of the API.
>
> Regarding checking system time,
> After reviewing the motivation of the KIP, it seems like a lower bound on how 
> long to suppress updates should be sufficient. This is reinforced by the fact 
> that the proposed behavior is to emit only when processing new data. Since 
> this is the proposed behavior, it should be fine to use the system time we 
> already checked at the start of the processing loop. Practically speaking, a 
> "best effort lower bound"-type guarantee might be a good starting point. It 
> gives us the flexibility to implement it efficiently, and we can always 
> tighten the bound later, if there are requests to do so.
>
> Regarding flushing on shutdown, can you elaborate of the motivation for doing 
> so?
>
> Thanks,
> -John
>
> On Mon, Mar 11, 2019 at 3:13 PM Matthias J. Sax  wrote:
>>
>> I agree that there are multiple ways how to avoid calling
>> `System.currentTimeMillis()`. However, the KIP needs to define the
>> public contract to explain users what behavior they can expect (the
>> simplest thing might be to say, it's based on `punctuation()` schedule
>> -- not sure if this is desired or not).
>>
>> Similarly, the question about "why should we slush on shutdown" is part
>> of the contract and multiple ways how to design it seem possible.
>>
>>
>>
>> -Matthias
>>
>> On 3/11/19 8:30 AM, John Roesler wrote:
>> > Hey, all, just chiming in to keep the discussion moving...
>> >
>> > Regarding whether to flush or not on shutdown, I'm curious why we *would*
>> > flush...
>> > The record cache does this, but that's because it's not durable. The
>> > suppression buffer is already backed by a changelog specifically so that it
>> > can provide exactly the timing you configure, and not have to emit early
>> > just because the commit interval is short or the task is migrated. So,
>> > regardless of the commit interval or application lifecycle, if I tell
>> > suppression to wait 5 minutes before emitting, it'll wait 5 minutes. It
>> > seems asymmetric for wall-clock suppression to behave differently.
>> >
>> > Regarding checking wall-clock time, yes, it can be expensive, but there are
>> > a number of ways we can cope with it without introducing a complicated
>> > algorithm:
>> > * use nano time
>> > * check the wall-clock once per batch and set it on the processor context
>> > in org.apache.kafka.streams.processor.internals.StreamThread#runOnce (we
>> > already check system time here anyway)
>> > * maybe just do the naive thing and measure the overhead. I.e., maybe we
>> > should benchmark the implementation anyway to look for this or other
>> > bottlenecks, and fix performance problem in the order they appear.
>> >
>> > Thoughts?
>> >
>> > Thanks,
>> > -John
>> >
>> > On Mon, Feb 25, 2019 at 4:36 PM jonathangor...@newrelic.com <
>> > jonathangor...@newrelic.com> wrote:
>> >
>> >> On 2019/02/21 02:19:27, "Matthias J. Sax"  wrote:
>> >>> thanks for the KIP. Corner case question:
>> >>>
>> >>> What happens if an application is stopped an restarted?
>> >>>
>> >>>  - Should suppress() flush all records (would be _before_ the time
>> >> elapsed)?
>> >>>  - Or should it preserve buffered records and reload on restart? For
>> >>> this case, should the record be flushed on reload (elapsed time is
>> >>> unknown) or should we reset the timer to zero?
>> >>
>> >> My opinion is that we should aim for simplicity for the first
>> >> implementation of this feature: Flush all the records on shutdown. If
>> >> there's demand in the future for strict adherence on shutdown we can
>> >> implement them as extra params to Suppressed api.
>> >>
>> >>> What is unclear to me atm, is the use-case you anticipate. If you assume
>> >>> a live run of an applications, event-time and processing-time should be
>> >>> fairly identical (at least with regard to data rates). Thus, suppress()
>> >>> on event-time should give you about the same behavior as wall-clock
>> >>> time? If you disagree, can you elaborate?
>> >>
>> >> Imagine a session window where you aggregate 10K events that usually occur
>> >> within 2-3 seconds of each other (event time). However, they are ingested
>> >> in batches of 1000 or so, spread out over 2-3 minutes (ingest time), and
>> >> not necessarily in order. It's important for us to be able to publish this
>> >> aggregate in real-time as we get new data (every 10 seconds) to keep our
>> >> time to glass low, but our data store is non-updateable so we'd like to
>> >> limit the number of aggregates we publish.
>> >>
>> >> If you imagine a case where all the event batches arrive in reverse order
>> >> 

Re: [Discuss] - KIP-532 - Add KStream#toTable to the Streams DSL

[aishwarya kumar]

Hello John,

Thank you for pointing this out to me, to maintain consistency across API's
it does make sense to allow users to define custom names for
their processors.

I've made the change in the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-523%3A+Add+KStream%23toTable+to+the+Streams+DSL

Best,
Aishwarya

On Tue, Sep 24, 2019 at 11:54 AM John Roesler  wrote:

> Hey Aishwarya,
>
> Thanks for the KIP! It looks good to me, although in a post-KIP-307
> world, we also need a "Named" parameter (to give the processor node a
> name, as opposed to the store itself).
>
> This would result in a total of four overloads:
> 1. no args
> 2. Named
> 3. Materialized
> 4. Materialized, Named
>
> I'd like to propose a re-design of the DSL in the future to clean this
> up, but for now, this is the pattern we have to follow.
>
> Thoughts?
>
> Thanks,
> -John
>
> On Tue, Sep 24, 2019 at 9:54 AM aishwarya kumar 
> wrote:
> >
> > Thank you for the suggestion Matthais, i've made the necessary changes in
> > the KIP.
> >
> > Keeping this thread open for further input.
> > KIP link:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-523%3A+Add+KStream%23toTable+to+the+Streams+DSL
> >
> > Best,
> > Aishwarya
> >
> > On Thu, Sep 19, 2019 at 10:50 AM aishwarya kumar 
> wrote:
> >
> > > Thanks Matthias,
> > >
> > > That does make sense, let me update the KIP to reflect the
> Materialization
> > > scenario.
> > >
> > > Best,
> > > Aishwarya
> > >
> > > On Tue, Sep 17, 2019, 2:49 PM Matthias J. Sax 
> > > wrote:
> > >
> > >> Aishwarya,
> > >>
> > >> thanks for the KIP. Overall, I think it makes sense to allow
> converting
> > >> a KStream into a KTable.
> > >>
> > >> From the KIP:
> > >>
> > >> > materializing these KTables should only be allowed if the overloaded
> > >> function with Materialized is used (and if optimization is turned on
> it may
> > >> still be only logically materialized if the queryable name is not
> set).
> > >>
> > >> Can you elaborate? I think the behavior we want should align with the
> > >> behavior of `StreamsBuilder#table()`.
> > >>
> > >> From my understanding (correct me if I am wrong) it should be:
> > >>
> > >> (1) If optimization is turned off, the KTable will always be
> > >> materialized, independent which method is used. The KTable will not be
> > >> queryable though.
> > >>
> > >> (2) If optimization is turned on and if `toTable()` is used, the
> KTable
> > >> may or may not be materialized. For this case, even if the KTable is
> > >> materialized, the store would not be queryable.
> > >>
> > >> (3) If `toTable(Materialized)` is use and a `storeName` or
> > >> `StoreSupplier` is specified, the store will always be materialized
> and
> > >> also be queryable. Otherwise, case (1) or (2) applies.
> > >>
> > >>
> > >>
> > >> -Matthias
> > >>
> > >>
> > >> On 9/17/19 6:42 AM, aishwarya kumar wrote:
> > >> > Hi All,
> > >> >
> > >> > Keeping this thread alive!!
> > >> >
> > >> > The aim is to add two methods Kstream.toTable() &
> > >> > Kstream.toTable(Materialized), so users can choose to convert
> their
> > >> > event stream into a changelog stream at any stage.
> > >> > wiki link :
> > >> >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-523:+Add+KStream%23toTable+to+the+Streams+DSL
> > >> > jira ticket : https://issues.apache.org/jira/browse/KAFKA-7658
> > >> >
> > >> > Best,
> > >> > Aishwarya
> > >> >
> > >> > On Fri, Sep 13, 2019 at 10:49 AM aishwarya kumar <
> ash26...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> Hello,
> > >> >>
> > >> >> Starting this thread to discuss KIP-532:
> > >> >> wiki link :
> > >> >>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-523:+Add+KStream%23toTable+to+the+Streams+DSL
> > >> >> jira ticket : https://issues.apache.org/jira/browse/KAFKA-7658
> > >> >>
> > >> >> There has been some discussion around the use-case of this KIP in
> the
> > >> Jira
> > >> >> ticket.
> > >> >>
> > >> >> Regards,
> > >> >> Aishwarya
> > >> >>
> > >> >
> > >>
> > >>
>

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Tom Bentley]

On Thu, Sep 26, 2019 at 2:16 PM Jun Rao  wrote:

> [...]
> 100. It would be useful to think through how to support SASL scram in the
> new world. If the only broker listener is SASL scram, we need to bootstrap
> the scram credential in each broker before we can start the broker.
> Currently, this can be done by adding the scram credential in ZK first and
> then start the brokers. It's not clear how this works when the metadata
> service is part of the brokers.
> [...]
>

Without wishing to derail the discussion here, I wrote KIP-506 a while ago
which raises some other issues with setting SCRAM credentials via the Admin
interface, though I admit I overlooked the bootstrapping problem. This
makes me wonder whether trying to set SCRAM credentials via the Admin
interface is the wrong approach. Perhaps a tool which could be run within
the controller quorum (similar to the existing script) is the only way to
provide equivalent security.

Cheers,

Tom

Re: Status of user Admin API for quotas

[Tom Bentley]

Hi Viktor and Colin,

Thanks for the update. Viktor, if you publish your KIP after summit then we
can at least what comes out of the discussion. Distinguishing between
normal ISR traffic and reassignment traffic would be nice (if that's
something your KIP would enable), and any such distinction would need to be
made in the API.

Kind regards,

Tom

On Thu, Sep 26, 2019 at 6:46 PM Viktor Somogyi-Vass 
wrote:

> Hi Tom, Colin,
>
> I abandoned my KIP because at that time I haven't had time unfortunately to
> continue working on it (KIP-248) and I wanted to rework it to remove the
> java based client since the admin commands use Scala. Since at that time it
> mostly had consensus on the design I think it might be a good approach to
> continue. It was still on my to-dos but didn't want to interrupt KIP-422
> because at that time it seemed to be active.
> If you're writing the email with the attempt of continuing then I can only
> endorse you or perhaps if you need someone to help in I could be able to
> join efforts (even by reviewing and participating the discussion).
>
> My KIP didn't include reassignment quotas but I've been working on a KIP
> about reworking replication throttling to introduce reassignment throttling
> so perhaps it would make to discuss them somewhat together. I'll try to
> publish it soon but not sure I can until after the summit.
>
> Viktor
>
> On Fri, Sep 20, 2019 at 6:30 PM Colin McCabe  wrote:
>
> > Hi Tom,
> >
> > As you said, there were a few KIPs, but they seem to have become
> inactive.
> >
> > It's kind of a tough problem-- probably at least as complex as the admin
> > interface for ACLs.
> >
> > There's also the headache of how reassignment quotas should work.  We
> > probably want to change that quota to actually throttle only reassignment
> > traffic, not just any non-ISR traffic as it does now.  Or add a different
> > quota type?
> >
> > best,
> > Colin
> >
> >
> > On Fri, Sep 20, 2019, at 04:38, Tom Bentley wrote:
> > > Hi,
> > >
> > > I was wondering, what is the current status of efforts to add an
> > > AdminClient API for managing user quotas? My trawl of the list archives
> > > didn't turn up anything current (KIP-248 was rejected KIP-422 was
> > > discussed), but perhaps I missed something.
> > >
> > > Many thanks,
> > >
> > > Tom
> > >
> >
>

[jira] [Created] (KAFKA-8954) Topic existence check is wrongly implemented in the DeleteOffset API

[David Jacot (Jira)]

David Jacot created KAFKA-8954:
--

 Summary: Topic existence check is wrongly implemented in the 
DeleteOffset API
 Key: KAFKA-8954
 URL: https://issues.apache.org/jira/browse/KAFKA-8954
 Project: Kafka
  Issue Type: Improvement
Reporter: David Jacot


The current DeleteOffset API check relies on the consumer group's committed 
offsets to decide if a topic exists or not. While this works in most of the 
cases, it does not work when a topic exists but it does not have any committed 
offsets yet. Moreover, it is not consistent with other APIs which rely on the 
metadata cache so it must be updated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-8953) Consider renaming `UsePreviousTimeOnInvalidTimestamp` timestamp extractor

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-8953:
--

 Summary: Consider renaming `UsePreviousTimeOnInvalidTimestamp` 
timestamp extractor
 Key: KAFKA-8953
 URL: https://issues.apache.org/jira/browse/KAFKA-8953
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


Kafka Streams ships couple of different timestamp extractors, one named 
`UsePreviousTimeOnInvalidTimestamp`.

Given the latest improvements with regard to time tracking, it seems 
appropriate to rename this class to `UsePartitionTimeOnInvalidTimestamp`, as we 
know have fixed definition of partition time, and also pass in partition time 
into the `#extract(...)` method, instead of some non-well-defined "previous 
timestamp".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-8952) Vulnerabilities found for jackson-databind-2.9.9.jar and guava-20.0.jar in latest Apache-kafka latest version 2.3.0

[Namrata Kokate (Jira)]

Namrata Kokate created KAFKA-8952:
-

 Summary: Vulnerabilities found for jackson-databind-2.9.9.jar and 
guava-20.0.jar in latest Apache-kafka latest version 2.3.0
 Key: KAFKA-8952
 URL: https://issues.apache.org/jira/browse/KAFKA-8952
 Project: Kafka
  Issue Type: New Feature
Affects Versions: 2.3.0
Reporter: Namrata Kokate


I am currently using apache kafka latest version-2.3.0, however When I deployed 
the binary on the containers, I can see the vulnerability reported for the two 
jars - jackson-databind-2.9.9.jar and  guava-20.0.jar
 
I can see these vulnerabilities have been removed in the 
jackson-databind-2.9.10.jar and guava-24.1.1-jre.jar jars but the apache-kafka 
version 2.3.0 does not include these new jars.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)