Hello all,
We want to kick off some discussions about error handling and logging
conventions. With a number of great patch contributions to Kafka recently,
it is good time for us to sit down and think a little bit more about the
coding style guidelines we have
core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala
ab60e9b3a4d063c838bdc7f97b3ac7d2ede87072
core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5
Diff: https://reviews.apache.org/r/24676/diff/
Testing
---
Thanks,
Guozhang
)
---
Unit tests
Thanks,
Guozhang Wang
/
On Tue, Aug 26, 2014 at 6:22 PM, Guozhang Wang wangg...@gmail.com
wrote:
Hello all,
We want to kick off some discussions about error handling and logging
conventions. With a number of great patch contributions to Kafka
recently
ce06d2c381348deef8559374869fcaed923da1d1
Diff: https://reviews.apache.org/r/25155/diff/
Testing
---
Thanks,
Guozhang Wang
/RequestPurgatory.scala
ce06d2c381348deef8559374869fcaed923da1d1
Diff: https://reviews.apache.org/r/25155/diff/
Testing
---
Thanks,
Guozhang Wang
/kafka/server/LeaderElectionTest.scala
https://reviews.apache.org/r/25136/#comment90406
For resulted maps that are used in the constructor parameters, as long as
the constructor parameter will not change we can use mapValues.
- Guozhang Wang
On Aug. 28, 2014, 2:28 a.m., Mayuresh Gharat wrote
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25044/#review51803
---
Ship it!
Thanks for the patch. LGTM.
- Guozhang Wang
On Aug. 26
On Aug. 28, 2014, 5:36 p.m., Guozhang Wang wrote:
Thanks for the patch, some general comments:
1. In general we would like to avoid using ._1 and ._2 simply due to
clarity of the code; instead we can use { case (key, value) = }.
2. After thinking about it twice, I think even
Changing
mapValues to map since we do not need to leave comments indicating code
change, but just comment on the purpose of coding. We can generally say sth.
like Create a new collection with map since it (maybe modified outside this
block / in another function call, etc).
- Guozhang Wang
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25155/#review51850
---
On Aug. 28, 2014, 5:12 p.m., Guozhang Wang wrote:
---
This is an automatically
, 5 p.m., Guozhang Wang wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/
---
(Updated Aug. 27, 2014, 5 p.m.)
Review
/ServerShutdownTest.scala
ab60e9b3a4d063c838bdc7f97b3ac7d2ede87072
core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5
Diff: https://reviews.apache.org/r/24676/diff/
Testing
---
Unit tests
Thanks,
Guozhang Wang
/scala/unit/kafka/server/ServerShutdownTest.scala
ab60e9b3a4d063c838bdc7f97b3ac7d2ede87072
core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5
Diff: https://reviews.apache.org/r/24676/diff/
Testing
---
Unit tests
Thanks,
Guozhang Wang
---
Thanks,
Guozhang Wang
/scala/unit/kafka/server/ServerShutdownTest.scala
ab60e9b3a4d063c838bdc7f97b3ac7d2ede87072
core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5
Diff: https://reviews.apache.org/r/24676/diff/
Testing
---
Unit tests
Thanks,
Guozhang Wang
The semantic of the offset API is to return the latest possible offset of
the message that is appended no later than the given timestamp. For
implementation, it will get the starting offset of the log segment that is
created no later than the given timestamp, and hence if your log segment
contains
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/#review52044
---
On Sept. 2, 2014, 8:37 p.m., Guozhang Wang wrote:
---
This is an automatically
/kafka/server/RequestPurgatoryTest.scala
168712de241125982d556c188c76514fceb93779
Diff: https://reviews.apache.org/r/25155/diff/
Testing
---
Thanks,
Guozhang Wang
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/#review52234
---
Ship it!
Ship It!
- Guozhang Wang
On Sept. 3, 2014, 6:27 p.m
---
On Sept. 2, 2014, 8:37 p.m., Guozhang Wang wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/
---
(Updated Sept. 2
---
On Sept. 2, 2014, 8:37 p.m., Guozhang Wang wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/
---
(Updated Sept
Just made a pass over the unresolved tickets tagged for 0.8.2, I think many
of them can be pushed to 0.8.3 / 0.9.
On Wed, Sep 3, 2014 at 8:05 PM, Jonathan Weeks jonathanbwe...@gmail.com
wrote:
+1 on a 0.8.1.2 release as described.
I manually applied patches to cobble together a working
---
Thanks,
Guozhang Wang
Thanks,
Guozhang Wang
On Sept. 9, 2014, 1:38 a.m., Joel Koshy wrote:
core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala, line 125
https://reviews.apache.org/r/25136/diff/4/?file=675768#file675768line125
I don't think this is required, since we are subsequently doing a map
over the view
Hi Steve:
1. the new producer will be included in the 0.8.2 release for production
usage.
2. The error you reported has been changed as a WARN in the latest trunk.
We realize this should not really be an error case. In general, any errors
will be propagated to the producer in the following two
, 2014, at 8:55 PM, Guozhang Wang wangg...@gmail.com wrote:
Hi Steve:
1. the new producer will be included in the 0.8.2 release for production
usage.
2. The error you reported has been changed as a WARN in the latest trunk.
We realize this should not really be an error case. In general, any
/kafka/utils/ZkUtils.scala
https://reviews.apache.org/r/25420/#comment92288
Can we rename to consumerOffsetForPartitionDir for naming consistency?
- Guozhang Wang
On Sept. 7, 2014, 7:22 p.m., Viktor Tarananeko wrote
as a view.
Mayuresh, could you submit another patch following this suggestion and also
incorporating Neha's comments?
- Guozhang Wang
On Sept. 3, 2014, 6:27 p.m., Mayuresh Gharat wrote:
---
This is an automatically generated e-mail
Hi Jacky,
Could you elaborate a bit on your use cases, like why you want to manually
truncate logs? Kafka provide a set of configs for data retention based on
data size and time (for example maintaining as much as 100 GB or up to 7
days of old data), would that be sufficient to you?
Guozhang
Hi,
In the new (Java) producer, you can pass in a callback function in the
FutureRecordMetadata send(ProducerRecord record, Callback callback)
call, which will be triggered when the ack is received.
Alternatively, you can also call Future.get() on the returned future
metadata, which will block
catching Throwable and printing stack trace.
- Guozhang Wang
On Sept. 23, 2014, 5:48 p.m., Mayuresh Gharat wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25944
Apache header missing.
- Guozhang Wang
On Sept. 24, 2014, 4:26 p.m., Jiangjie Qin wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25995
https://reviews.apache.org/r/26306/#comment95770
Can we change this comment accordingly, since we have other places calling
resumeTopicDeletionThread() than these three cases.
- Guozhang Wang
On Oct. 3, 2014, 1:31 a.m., Sriharsha Chintalapani wrote
---
Thanks,
Guozhang Wang
---
Thanks,
Guozhang Wang
for logging,
could we just merge it into the second check?
- Guozhang Wang
On Oct. 6, 2014, 5:06 p.m., Jiangjie Qin wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26373
/25995/#comment96027
queueNumItemCapacity and queueByteCapacity?
- Guozhang Wang
On Oct. 6, 2014, 5:20 p.m., Jiangjie Qin wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25995
On Oct. 7, 2014, 12:15 a.m., Guozhang Wang wrote:
Since now the first iteration of if statements is only used for logging,
could we just merge it into the second check?
Jiangjie Qin wrote:
Guozhang, thanks for the review. I actually thought about it before. I
agree that the code
://reviews.apache.org/r/26390/diff/
Testing
---
Thanks,
Guozhang Wang
Hello all,
I put some thoughts on enhancing our current message metadata format to
solve a bunch of existing issues:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Enriched+Message+Metadata
This wiki page is for kicking off some discussions about the feasibility of
adding more info
---
Unit tests
Thanks,
Guozhang Wang
and the mailing list. Personally, I think it is better to discuss on
the mailing list (for more visibility) and just post a bold link to
the (archived) mailing list thread on the wiki.
Joel
On Fri, Oct 10, 2014 at 05:33:52PM -0700, Guozhang Wang wrote:
Hello all,
I put some thoughts
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26373/#review56736
---
Ship it!
Ship It!
- Guozhang Wang
On Oct. 13, 2014, 11:38 p.m
encoders/decoders but in broker by topic regardless of producer) to the
topics that have it enabled. We should have these maintained in the project
under contrib.
=- Joestein
On Mon, Oct 13, 2014 at 11:02 PM, Guozhang Wang wangg...@gmail.com
wrote:
Hi Jay,
Thanks for the comments. Replied
about, I suggest we take it out,
think
it end to end and then include it in 0.8.3.)
https://issues.apache.org/jira/browse/KAFKA-1634 (This has an owner:
Guozhang Wang)
https://issues.apache.org/jira/browse/KAFKA-1671 (Has a patch and is
waiting on a review by Joe Stein)
It seems
days.
Thanks,
Jun
On Thu, Oct 16, 2014 at 3:01 PM, Guozhang Wang wangg...@gmail.com wrote:
Regarding 1634, I was intended to work on that after 1583 since it will
changes the commit offset request handling logic a lot. If people think
1583 is only a few days away before check-in, we can
---
On Oct. 14, 2014, 2:42 a.m., Guozhang Wang wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676
/server/ServerShutdownTest.scala
3804a114e97c849cae48308997037786614173fc
core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5
Diff: https://reviews.apache.org/r/24676/diff/
Testing
---
Unit tests
Thanks,
Guozhang Wang
; with this we do not need to add another logic
for leadership map and checkpoint file, but just the logic on
replica-manager to handle this extra controlled message and remembering the
current leader epoch instead of a map.
Thanks,
Jun
On Fri, Oct 10, 2014 at 5:33 PM, Guozhang Wang wangg
-compression on brokers and MMs with log
compaction turned on.
3. How we can resolve unclean leader election resulted data inconsistency
with control messages.
Guozhang
On Sun, Oct 19, 2014 at 11:41 PM, Guozhang Wang wangg...@gmail.com wrote:
Thanks for the detailed comments Jun! Some replies inlined
Is this change intended?
- Guozhang Wang
On Oct. 21, 2014, 8:37 p.m., Jiangjie Qin wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26994
, Guozhang Wang wangg...@gmail.com wrote:
I have updated the wiki page incorporating received comments. We can
discuss some more details on:
1. How we want to do audit? Whether we want to have in-built auditing on
brokers or even MMs or use an audit consumer to fetch all messages from
just
/server/ServerShutdownTest.scala
3804a114e97c849cae48308997037786614173fc
core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5
Diff: https://reviews.apache.org/r/24676/diff/
Testing
---
Unit tests
Thanks,
Guozhang Wang
:
https://reviews.apache.org/r/24676/#review57235
---
On Oct. 23, 2014, 1:53 a.m., Guozhang Wang wrote:
---
This is an automatically generated e-mail. To reply, visit:
https
On Oct. 22, 2014, 9:32 p.m., Neha Narkhede wrote:
core/src/main/scala/kafka/tools/MirrorMaker.scala, line 271
https://reviews.apache.org/r/26994/diff/1/?file=727975#file727975line271
Is there any value in setting this to true? It seems that just checking
if it is false and
/browse/KAFKA-1501
Repository: kafka
Description
---
needs repeated unit tests to verify
Diffs
-
core/src/main/scala/kafka/network/SocketServer.scala
cee76b323e5f3e4c783749ac9e78e1ef02897e3b
Diff: https://reviews.apache.org/r/27101/diff/
Testing
---
Thanks,
Guozhang Wang
and exiting the process suffices. Setting to true something
that is called cleanShutdown, when in fact, it isn't a clean shutdown is
confusing to read.
Also good to add a FATAL log entry as suggested by Guozhang as well.
Guozhang Wang wrote:
The boolean is used when the internal
/scala/kafka/tools/MirrorMaker.scala
https://reviews.apache.org/r/26994/#comment99094
fatal(Consumer thread failure due to , t)
- Guozhang Wang
On Oct. 23, 2014, 11:20 p.m., Jiangjie Qin wrote:
---
This is an automatically generated e
Jack,
The fetchers are blocked on the queue since it is full, is your consumer
iterator stopped and hence not getting more data from it?
Guozhang
On Thu, Oct 23, 2014 at 3:53 PM, Jack Foy j...@whitepages.com wrote:
Hi all,
We run kafka 0.8.1.1. We’re tracking down a problem where consumer
: https://reviews.apache.org/r/27214/diff/
Testing
---
Thanks,
Guozhang Wang
?
- Guozhang Wang
On Oct. 23, 2014, 11:19 p.m., Ewen Cheslack-Postava wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26885
Hello folks,
I came across this testComplexCompressDecompress in
kafka.message.MessageCompressionTest while I'm working some consumer
decompression optimization. This test checks if nested compression is
supported.
I remember vaguely that some time ago we decide not to support nested
compression
---
Thanks,
Guozhang Wang
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27256/#review58744
---
On Oct. 27, 2014, 11:59 p.m., Guozhang Wang wrote
3804a114e97c849cae48308997037786614173fc
core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5
Diff: https://reviews.apache.org/r/24676/diff/
Testing
---
Unit tests
Thanks,
Guozhang Wang
/RequestResponseSerializationTest.scala
cd16ced5465d098be7a60498326b2a98c248f343
Diff: https://reviews.apache.org/r/27391/diff/
Testing
---
Thanks,
Guozhang Wang
Thanks Jun and Neha. I will go ahead and file the JIRA to remove this test
case then.
cd3fc142176b8a3638db5230fb74f055c04c59d4
Diff: https://reviews.apache.org/r/27430/diff/
Testing
---
Thanks,
Guozhang Wang
p.m., Guozhang Wang wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/
---
(Updated Oct. 28, 2014, 10:09 p.m.)
Review
system_test/metrics.json cd3fc142176b8a3638db5230fb74f055c04c59d4
Diff: https://reviews.apache.org/r/27430/diff/
Testing
---
Thanks,
Guozhang Wang
?
- Guozhang Wang
On Nov. 3, 2014, 7:46 p.m., Ewen Cheslack-Postava wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27534
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27535/#review59899
---
Ship it!
Thanks for the patch. LGTM.
- Guozhang Wang
On Nov. 3
On Nov. 5, 2014, 1 a.m., Guozhang Wang wrote:
Where will testcaseEnv.validationStatusDict[Test completed] be used?
Ewen Cheslack-Postava wrote:
That's where all the validation results (the test's assertions) are
stored. It gets scanned through at the end of the test run
cd16ced5465d098be7a60498326b2a98c248f343
core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala
8c5364fa97da1be09973c176d1baeb339455d319
Diff: https://reviews.apache.org/r/27391/diff/
Testing
---
Thanks,
Guozhang Wang
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27723/#review60359
---
Ship it!
- Guozhang Wang
On Nov. 7, 2014, 12:26 p.m., Manikumar
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27391/#review60285
---
On Nov. 6, 2014, 11:35 p.m., Guozhang Wang wrote
constructor without timestamp and
add a secondary constructor with timestamp and mark deprecated there.
Also, can we use case class.copy if timestamp needs to be modified?
However, per comment further down I don't think it needs to be touched.
Guozhang Wang wrote:
Actually we cannot
8c5364fa97da1be09973c176d1baeb339455d319
Diff: https://reviews.apache.org/r/27391/diff/
Testing
---
Thanks,
Guozhang Wang
Hi Alan,
The reason we do not have per-topic parallelism spec in wildcard is two
folds: 1) we use a per-topic hash-based partition algorithm, and hence
having each topic with the same num. of streams may give us better load
balance, 2) with the topicFilter we will not know exactly which topics to
Joe,
Are you looking for this?
/** coderetries/code */
public static final String RETRIES_CONFIG = retries;
Guozhang
On Mon, Nov 10, 2014 at 8:11 AM, Joe Stein joe.st...@stealth.ly wrote:
In the new Java producer I see that you can set RETRY_BACKOFF_MS_CONFIG
Thanks Jun!
Jiangjie, could you file a JIRA? Thanks.
Guozhang
On Tue, Nov 11, 2014 at 9:27 AM, Jun Rao jun...@gmail.com wrote:
Hi, Jiangjie,
Thanks for the investigation. Yes, this seems like a real issue.
1. It doesn't seem that we need to put the shutdownCommand back into the
queue.
the shutdown command?
- Guozhang Wang
On Nov. 11, 2014, 10:59 p.m., Jiangjie Qin wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27890
.
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/
On Mon, Oct 20, 2014 at 2:15 AM, Guozhang Wang wangg...@gmail.com
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27890/#review61099
---
Ship it!
Ship It!
- Guozhang Wang
On Nov. 12, 2014, 10:05 p.m
the record to an underlying
KafkaProducer. We can add a similar ConsumerFactory to the new consumer
client. If a user plugs in an implementation of the AuditingConsumer, the
consumer will then be audited automatically.
Thanks,
Jun
On Tue, Oct 21, 2014 at 4:06 PM, Guozhang Wang wangg
Hi Dibyendu,
Yes we are changing the consumer API in 0.9, which will be different with
the current high-level consumer API. We are also trying to figure out a way
to preserve the functionality: KAFKA-1655
https://issues.apache.org/jira/browse/KAFKA-1655
Guozhang
On Fri, Nov 14, 2014 at 5:29 PM,
Just added an entry in the FAQ page:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tIspecifythenumberofstreamsparallelismpertopicmapusingwildcardstreamasIusestaticstreamhandler
?
On Mon, Nov 10, 2014 at 7:56 AM, Guozhang Wang wangg...@gmail.com wrote:
Hi Alan,
The reason
sending messages
2. Broker does not have the specified partition by id upon receiving messages
core/src/main/scala/kafka/common/UnknownTopicOrPartitionException.scala
https://reviews.apache.org/r/28040/#comment103633
remove (and the end on line 23)
- Guozhang Wang
On Nov. 14, 2014, 4
thread exits due to
...
which is a bit verbose?
- Guozhang Wang
On Nov. 12, 2014, 5:51 p.m., Jiangjie Qin wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25995
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27391/#review61761
---
On Nov. 8, 2014, 12:54 a.m., Guozhang Wang wrote
this timestamp? i.e., we should
basically ignore it right? So we only need to do:
value.set(VALUE_TIMESTAMP_FIELD, expirationTimestamp).
Guozhang Wang wrote:
I think note. In v0/v1, if the timestamp is explicitly specified (i.e.
not -1) we need to use it as the expiration
/test/scala/unit/kafka/server/OffsetCommitTest.scala
8c5364fa97da1be09973c176d1baeb339455d319
Diff: https://reviews.apache.org/r/27391/diff/
Testing
---
Thanks,
Guozhang Wang
/ProducerFailureHandlingTest.scala
8531f533f3a6431f4f9fc8cb1ad3e9f1f1b110e0
Diff: https://reviews.apache.org/r/28240/diff/
Testing
---
Thanks,
Guozhang Wang
11eb11355efddacf62d61690ad13b9c82a200230
gradle.properties 5d3155fd4461438d8b2ec4faa9534cc2383d4951
scala.gradle 6adf9af7dbbe71e68a07b387c3854d8c9ad339e0
Diff: https://reviews.apache.org/r/28268/diff/
Testing
---
Thanks,
Guozhang Wang
everyone,
I'm very happy to announce that the Kafka PMC has invited Guozhang
Wang to
become a committer. Guozhang has made significant contributions to
Kafka
over the past year, along with being very active on code reviews and
the
mailing list.
Please join me in welcoming him
.
- Guozhang
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28240/#review62547
---
On Nov. 19, 2014, 7:01 p.m., Guozhang Wang wrote
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27391/#review62553
---
On Nov. 18, 2014, 1:42 a.m., Guozhang Wang wrote
.
- Guozhang
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27391/#review62553
---
On Nov. 18, 2014, 1:42 a.m., Guozhang Wang wrote
501 - 600 of 6729 matches
Mail list logo
