[jira] [Commented] (KAFKA-2602) Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and 0.8.3 documentation
[ https://issues.apache.org/jira/browse/KAFKA-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583514#comment-15583514 ] Andrew Musselman commented on KAFKA-2602: - [~hachikuji] [~ijuma] [~junrao] [~granthenke] any guidance whether this is valid/still an open issue? > Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and > 0.8.3 documentation > - > > Key: KAFKA-2602 > URL: https://issues.apache.org/jira/browse/KAFKA-2602 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.2.1, 0.9.0.0 >Reporter: Zhuo Liu >Priority: Minor > Labels: newbie > Fix For: 0.8.2.0 > > Attachments: KAFKA-2602_2015-09-30_14_22_22.patch > > > The documentation still says "serializer.class" and "key.serializer.class", > which is no longer correct. > We need to add descriptions about "key.serializer" and "value.serializer" in > the new Producer Config section (section 3.4). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3545) Generalized Serdes for List/Map
[ https://issues.apache.org/jira/browse/KAFKA-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583196#comment-15583196 ] Andrew Musselman commented on KAFKA-3545: - Not sure who to notifiy for closing tickets; [~gwenshap] could you suggest someone? > Generalized Serdes for List/Map > --- > > Key: KAFKA-3545 > URL: https://issues.apache.org/jira/browse/KAFKA-3545 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Greg Fodor >Priority: Minor > Labels: api, newbie > Fix For: 0.10.1.0 > > > In working with Kafka Streams I've found it's often the case I want to > perform a "group by" operation, where I repartition a stream based on a > foreign key and then do an aggregation of all the values into a single > collection, so the stream becomes one where each entry has a value that is a > serialized list of values that belonged to the key. (This seems unrelated to > the 'group by' operation talked about in KAFKA-3544.) Basically the same > typical group by operation found in systems like Cascading. > In order to create these intermediate list values I needed to define custom > avro schemas that simply wrap the elements of interest into a list. It seems > desirable that there be some basic facility for constructing simple Serdes of > Lists/Maps/Sets of other types, potentially using avro's serialization under > the hood. If this existed in the core library it would also enable the > addition of higher level operations on streams that can use these Serdes to > perform simple operations like the "group by" example I mention. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2111) Command Line Standardization - Add Help Arguments & List Required Fields
[ https://issues.apache.org/jira/browse/KAFKA-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583142#comment-15583142 ] Andrew Musselman commented on KAFKA-2111: - [~mwarhaftig] any progress on this; are we keeping this open as a placeholder still? > Command Line Standardization - Add Help Arguments & List Required Fields > > > Key: KAFKA-2111 > URL: https://issues.apache.org/jira/browse/KAFKA-2111 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Matt Warhaftig >Priority: Minor > Labels: newbie > > KIP-14 is the standardization of tool command line arguments. As an offshoot > of that proposal there are standardization changes that don't need to be part > of the KIP since they are less invasive. They are: > - Properly format argument descriptions (into sentences) and add any missing > "REQUIRED" notes. > - Add 'help' argument to any top-level tool scripts that were missing it. > This JIRA is for tracking them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3545) Generalized Serdes for List/Map
[ https://issues.apache.org/jira/browse/KAFKA-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583131#comment-15583131 ] Andrew Musselman commented on KAFKA-3545: - [~guozhang] and [~gfodor] looks like the two tickets you mention are resolved; can we close this one? > Generalized Serdes for List/Map > --- > > Key: KAFKA-3545 > URL: https://issues.apache.org/jira/browse/KAFKA-3545 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Greg Fodor >Priority: Minor > Labels: api, newbie > > In working with Kafka Streams I've found it's often the case I want to > perform a "group by" operation, where I repartition a stream based on a > foreign key and then do an aggregation of all the values into a single > collection, so the stream becomes one where each entry has a value that is a > serialized list of values that belonged to the key. (This seems unrelated to > the 'group by' operation talked about in KAFKA-3544.) Basically the same > typical group by operation found in systems like Cascading. > In order to create these intermediate list values I needed to define custom > avro schemas that simply wrap the elements of interest into a list. It seems > desirable that there be some basic facility for constructing simple Serdes of > Lists/Maps/Sets of other types, potentially using avro's serialization under > the hood. If this existed in the core library it would also enable the > addition of higher level operations on streams that can use these Serdes to > perform simple operations like the "group by" example I mention. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface
[ https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583079#comment-15583079 ] Andrew Musselman commented on KAFKA-3745: - Still keeping this open? > Consider adding join key to ValueJoiner interface > - > > Key: KAFKA-3745 > URL: https://issues.apache.org/jira/browse/KAFKA-3745 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.0.0 >Reporter: Greg Fodor >Priority: Minor > Labels: api, newbie > > In working with Kafka Stream joining, it's sometimes the case that a join key > is not actually present in the values of the joins themselves (if, for > example, a previous transform generated an ephemeral join key.) In such > cases, the actual key of the join is not available in the ValueJoiner > implementation to be used to construct the final joined value. This can be > worked around by explicitly threading the join key into the value if needed, > but it seems like extending the interface to pass the join key along as well > would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-189) Create a script for dump segment tool
[ https://issues.apache.org/jira/browse/KAFKA-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583075#comment-15583075 ] Andrew Musselman commented on KAFKA-189: Any interest in this; been open a while. > Create a script for dump segment tool > - > > Key: KAFKA-189 > URL: https://issues.apache.org/jira/browse/KAFKA-189 > Project: Kafka > Issue Type: New Feature > Components: core >Reporter: Taylor Gautier >Priority: Minor > Labels: newbie, patch > Attachments: KAFKA-189.diff > > > In the kafka jar there is a useful segment dump tool. It would be useful to > expose this as a script. The current usage is: > bin/kafka-run-class.sh kafka.tools.DumpLogSegments logfilename -noprint > Change this to: > bin/kafka-dump-log-segments.sh logfilename -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2424) Consider introducing lint-like tool for Scala
[ https://issues.apache.org/jira/browse/KAFKA-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583065#comment-15583065 ] Andrew Musselman commented on KAFKA-2424: - Looks like Abide doesn't have a way to include it in Gradle builds yet and Scapegoat doesn't even mention Gradle; any sense of when Abide will support Gradle? > Consider introducing lint-like tool for Scala > - > > Key: KAFKA-2424 > URL: https://issues.apache.org/jira/browse/KAFKA-2424 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma > Labels: newbie > > Typesafe is working on abide and the first release is expected next month: > https://github.com/scala/scala-abide > An alternative is scapegoat: > https://github.com/sksamuel/scalac-scapegoat-plugin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2607) Review `Time` interface and its usage
[ https://issues.apache.org/jira/browse/KAFKA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583043#comment-15583043 ] Andrew Musselman commented on KAFKA-2607: - Looks like the PR has conflicts now; is this still in progress or is it still blocked by KAFKA-2247? > Review `Time` interface and its usage > - > > Key: KAFKA-2607 > URL: https://issues.apache.org/jira/browse/KAFKA-2607 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.2.2 >Reporter: Ismael Juma > Labels: newbie > > Two of `Time` interface's methods are `milliseconds` and `nanoseconds` which > are implemented in `SystemTime` as follows: > {code} > @Override > public long milliseconds() { > return System.currentTimeMillis(); > } > @Override > public long nanoseconds() { > return System.nanoTime(); > } > {code} > The issue with this interface is that it makes it seem that the difference is > about the unit (`ms` versus `ns`) whereas it's much more than that: > https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks > We should probably change the names of the methods and review our usage to > see if we're using the right one in the various places. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2247) Merge kafka.utils.Time and kafka.common.utils.Time
[ https://issues.apache.org/jira/browse/KAFKA-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583038#comment-15583038 ] Andrew Musselman commented on KAFKA-2247: - [~aauradkar] [~ijuma] Do you have a path to resolve this? Looks like KAFKA-2607 is blocked by this but there's a patch for it, though it has conflicts now: https://github.com/apache/kafka/pull/837 > Merge kafka.utils.Time and kafka.common.utils.Time > -- > > Key: KAFKA-2247 > URL: https://issues.apache.org/jira/browse/KAFKA-2247 > Project: Kafka > Issue Type: Improvement >Reporter: Aditya Auradkar >Assignee: Aditya Auradkar >Priority: Minor > > We currently have 2 different versions of Time in clients and core. These > need to be merged. > It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is > used by some tests while `o.a.kafka.common.utils.Time` does not. We either > need to add this functionality or change the tests not to need it anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3408) consumer rebalance fail
[ https://issues.apache.org/jira/browse/KAFKA-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583027#comment-15583027 ] Andrew Musselman commented on KAFKA-3408: - [~sdhzlzhk] Can you provide more repro info? > consumer rebalance fail > --- > > Key: KAFKA-3408 > URL: https://issues.apache.org/jira/browse/KAFKA-3408 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.9.0.1 > Environment: centos linux >Reporter: zhongkai liu > Labels: newbie > > I use "/bin/kafka-console-consumer" command to start two consumers of group > "page_group",then the first conumer console report rebalance failure like > this: > ERROR [page_view_group1_slave2-1458095694092-80c33086], error during > syncedRebalance (kafka.consumer.ZookeeperConsumerConnector) > kafka.common.ConsumerRebalanceFailedException: > page_view_group1_slave2-1458095694092-80c33086 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660) > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:579) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1703) The bat script failed to start on windows
[ https://issues.apache.org/jira/browse/KAFKA-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583022#comment-15583022 ] Andrew Musselman commented on KAFKA-1703: - Has this been vetted/merged? > The bat script failed to start on windows > - > > Key: KAFKA-1703 > URL: https://issues.apache.org/jira/browse/KAFKA-1703 > Project: Kafka > Issue Type: Bug > Components: build >Affects Versions: 0.8.1.1 >Reporter: ChengRen > Labels: newbie > Attachments: kafka-run-class.bat > > > The bat script in bin\windows can not start zookeeper and kafka correctly > (where my os is just installed and only jdk ready). I modified the > kafka-run-class.bat and add jars in libs folder to classpath. > for %%i in (%BASE_DIR%\core\lib\*.jar) do ( > call :concat %%i > ) > added begin > for %%i in (%BASE_DIR%\..\libs\*.jar) do ( > call :concat %%i > ) > added end > for %%i in (%BASE_DIR%\perf\target\scala-2.8.0/kafka*.jar) do ( > call :concat %%i > ) > Now it runs correctly. > Under bin\windows: > zookeeper-server-start.bat ..\..\config\zookeeper.properties >kafka-server-start.bat ..\..\config\kafka.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1715) better advertising of the bound and working interface
[ https://issues.apache.org/jira/browse/KAFKA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582992#comment-15582992 ] Andrew Musselman commented on KAFKA-1715: - [~joestein] any more description/guidance, or is this not an issue any more? > better advertising of the bound and working interface > - > > Key: KAFKA-1715 > URL: https://issues.apache.org/jira/browse/KAFKA-1715 > Project: Kafka > Issue Type: Bug >Reporter: Joe Stein > Labels: newbie > > As part of the auto discovery of brokers and meta data messaging we should > try to advertise the interface that is bound and working better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2200) kafkaProducer.send() should not call callback.onCompletion()
[ https://issues.apache.org/jira/browse/KAFKA-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583016#comment-15583016 ] Andrew Musselman commented on KAFKA-2200: - Still an issue? > kafkaProducer.send() should not call callback.onCompletion() > > > Key: KAFKA-2200 > URL: https://issues.apache.org/jira/browse/KAFKA-2200 > Project: Kafka > Issue Type: Bug >Reporter: Jiangjie Qin > Labels: newbie > > KafkaProducer.send() should not call callback.onCompletion() because this > might break the callback firing order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
[ https://issues.apache.org/jira/browse/KAFKA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583009#comment-15583009 ] Andrew Musselman commented on KAFKA-2167: - Looks like this ticket should be closed. > ZkUtils updateEphemeralPath JavaDoc (spelling and correctness) > -- > > Key: KAFKA-2167 > URL: https://issues.apache.org/jira/browse/KAFKA-2167 > Project: Kafka > Issue Type: Bug >Reporter: Jon Bringhurst > Labels: newbie > > I'm not 100% sure on this, but it seems like "persistent" should instead say > "ephemeral" in the JavaDoc. Also, note that "parrent" is misspelled. > {noformat} > /** >* Update the value of a persistent node with the given path and data. >* create parrent directory if necessary. Never throw NodeExistException. >*/ > def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit > = { > try { > client.writeData(path, data) > } > catch { > case e: ZkNoNodeException => { > createParentPath(client, path) > client.createEphemeral(path, data) > } > case e2 => throw e2 > } > } > {noformat} > should be: > {noformat} > /** >* Update the value of an ephemeral node with the given path and data. >* create parent directory if necessary. Never throw NodeExistException. >*/ > def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit > = { > try { > client.writeData(path, data) > } > catch { > case e: ZkNoNodeException => { > createParentPath(client, path) > client.createEphemeral(path, data) > } > case e2 => throw e2 > } > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2602) Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and 0.8.3 documentation
[ https://issues.apache.org/jira/browse/KAFKA-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582987#comment-15582987 ] Andrew Musselman commented on KAFKA-2602: - Is this still open/should this patch be applied? > Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and > 0.8.3 documentation > - > > Key: KAFKA-2602 > URL: https://issues.apache.org/jira/browse/KAFKA-2602 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.2.1, 0.9.0.0 >Reporter: Zhuo Liu >Priority: Minor > Labels: newbie > Fix For: 0.8.2.0 > > Attachments: KAFKA-2602_2015-09-30_14_22_22.patch > > > The documentation still says "serializer.class" and "key.serializer.class", > which is no longer correct. > We need to add descriptions about "key.serializer" and "value.serializer" in > the new Producer Config section (section 3.4). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2544) Replication tools wiki page needs to be updated
[ https://issues.apache.org/jira/browse/KAFKA-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582980#comment-15582980 ] Andrew Musselman commented on KAFKA-2544: - [~gwenshap] do you know who could add Lauren to the wiki contributor list? > Replication tools wiki page needs to be updated > --- > > Key: KAFKA-2544 > URL: https://issues.apache.org/jira/browse/KAFKA-2544 > Project: Kafka > Issue Type: Improvement > Components: website >Affects Versions: 0.8.2.1 >Reporter: Stevo Slavic >Priority: Minor > Labels: documentation, newbie > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools is > outdated, mentions tools which have been heavily refactored or replaced by > other tools, e.g. add partition tool, list/create topics tools, etc. > Please have the replication tools wiki page updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1351) String.format is very expensive in Scala
[ https://issues.apache.org/jira/browse/KAFKA-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582973#comment-15582973 ] Andrew Musselman commented on KAFKA-1351: - Still open? > String.format is very expensive in Scala > > > Key: KAFKA-1351 > URL: https://issues.apache.org/jira/browse/KAFKA-1351 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.7.2, 0.8.0, 0.8.1 >Reporter: Neha Narkhede > Labels: newbie > Attachments: KAFKA-1351.patch, KAFKA-1351_2014-04-07_18:02:18.patch, > KAFKA-1351_2014-04-09_15:40:11.patch > > > As found in KAFKA-1350, logging is causing significant overhead in the > performance of a Kafka server. There are several info statements that use > String.format which is particularly expensive. We should investigate adding > our own version of String.format that merely uses string concatenation under > the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer
[ https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582896#comment-15582896 ] Andrew Musselman commented on KAFKA-4308: - It is; mark this as a dupe? > Inconsistent parameters between console producer and consumer > - > > Key: KAFKA-4308 > URL: https://issues.apache.org/jira/browse/KAFKA-4308 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.1.0 >Reporter: Gwen Shapira > Labels: newbie > > kafka-console-producer uses --broker-list while kafka-console-consumer uses > --bootstrap-server. > Let's add --bootstrap-server to the producer for some consistency? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-205) Audit swallowing of exceptions/throwables
[ https://issues.apache.org/jira/browse/KAFKA-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582871#comment-15582871 ] Andrew Musselman commented on KAFKA-205: Still a need for this audit? > Audit swallowing of exceptions/throwables > - > > Key: KAFKA-205 > URL: https://issues.apache.org/jira/browse/KAFKA-205 > Project: Kafka > Issue Type: Task >Reporter: Chris Burroughs > Labels: newbie > > There are several cases like KAFKA-204 where we swallow Error, or even > Throwable. This is probably not the right thing to do (is > java.lang.StackOverflowError > ever recoverable?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer
[ https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582846#comment-15582846 ] Andrew Musselman commented on KAFKA-4308: - Oh I just meant go ahead, hadn't seen your comment; I don't have permissions to assign tickets. > Inconsistent parameters between console producer and consumer > - > > Key: KAFKA-4308 > URL: https://issues.apache.org/jira/browse/KAFKA-4308 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.1.0 >Reporter: Gwen Shapira > Labels: newbie > > kafka-console-producer uses --broker-list while kafka-console-consumer uses > --bootstrap-server. > Let's add --bootstrap-server to the producer for some consistency? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer
[ https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582824#comment-15582824 ] Andrew Musselman commented on KAFKA-4308: - Ha, Umesh go for it. > Inconsistent parameters between console producer and consumer > - > > Key: KAFKA-4308 > URL: https://issues.apache.org/jira/browse/KAFKA-4308 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.1.0 >Reporter: Gwen Shapira > Labels: newbie > > kafka-console-producer uses --broker-list while kafka-console-consumer uses > --bootstrap-server. > Let's add --bootstrap-server to the producer for some consistency? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer
[ https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582822#comment-15582822 ] Andrew Musselman commented on KAFKA-4308: - I'll do it and make a PR if that's the right workflow. > Inconsistent parameters between console producer and consumer > - > > Key: KAFKA-4308 > URL: https://issues.apache.org/jira/browse/KAFKA-4308 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.1.0 >Reporter: Gwen Shapira > Labels: newbie > > kafka-console-producer uses --broker-list while kafka-console-consumer uses > --bootstrap-server. > Let's add --bootstrap-server to the producer for some consistency? -- This message was sent by Atlassian JIRA (v6.3.4#6332)