[jira] [Created] (KAFKA-7933) KTableKTableLeftJoinTest takes an hour to finish
Viktor Somogyi created KAFKA-7933:
-
Summary: KTableKTableLeftJoinTest takes an hour to finish
Key: KAFKA-7933
URL: https://issues.apache.org/jira/browse/KAFKA-7933
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 2.2.0
Reporter: Viktor Somogyi
Attachments: jenkins-output-one-hour-test.log
PRs might time out as
{{KTableKTableLeftJoinTest.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions}}
took one hour to complete.
{noformat}
11:57:45 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest >
shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions STARTED
12:53:35
12:53:35 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest >
shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions PASSED
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7823) Missing release notes pages
Viktor Somogyi created KAFKA-7823: - Summary: Missing release notes pages Key: KAFKA-7823 URL: https://issues.apache.org/jira/browse/KAFKA-7823 Project: Kafka Issue Type: Improvement Affects Versions: 1.1.0, 1.0.1, 1.0.0 Reporter: Viktor Somogyi These 3 pages are not available on the apache website, however other release are there (such as 2.0.0). https://www.apache.org/dist/kafka/1.1.0/RELEASE_NOTES.html https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html https://www.apache.org/dist/kafka/1.0.0/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7805) Use --bootstrap-server option in ducktape tests where applicable
Viktor Somogyi created KAFKA-7805:
-
Summary: Use --bootstrap-server option in ducktape tests where
applicable
Key: KAFKA-7805
URL: https://issues.apache.org/jira/browse/KAFKA-7805
Project: Kafka
Issue Type: Improvement
Components: system tests
Reporter: Viktor Somogyi
KIP-377 introduces the {{--bootstrap-server}} option and deprecates the
{{--zookeeper}} option in {{kafka-topics.sh}}. I'd be nice to use the new
option in the ducktape tests to gain higher test coverage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7804) Update the docs for KIP-377
Viktor Somogyi created KAFKA-7804:
-
Summary: Update the docs for KIP-377
Key: KAFKA-7804
URL: https://issues.apache.org/jira/browse/KAFKA-7804
Project: Kafka
Issue Type: Improvement
Components: documentation
Reporter: Viktor Somogyi
Assignee: Viktor Somogyi
KIP-377 introduced the {{--bootstrap-server}} option to the {{kafka-topics.sh}}
command. The documentation (examples) should be updated accordingly and a
release note should be added,
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7737) Consolidate InitProducerId API
Viktor Somogyi created KAFKA-7737: - Summary: Consolidate InitProducerId API Key: KAFKA-7737 URL: https://issues.apache.org/jira/browse/KAFKA-7737 Project: Kafka Issue Type: Task Components: producer Reporter: Viktor Somogyi Assignee: Viktor Somogyi We have two separate paths in the producer for the InitProducerId API: one for the transactional producer and one for the idempotent producer. It would be nice to find a way to consolidate these. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7736) Consolidate Map usages in TransactionManager
Viktor Somogyi created KAFKA-7736: - Summary: Consolidate Map usages in TransactionManager Key: KAFKA-7736 URL: https://issues.apache.org/jira/browse/KAFKA-7736 Project: Kafka Issue Type: Task Components: producer Reporter: Viktor Somogyi Assignee: Viktor Somogyi There are a bunch of Map collections in TransactionManager which could be consolidated into a single map to consolidate bookkeeping and get rid of potential bugs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7617) Document security primitives
Viktor Somogyi created KAFKA-7617: - Summary: Document security primitives Key: KAFKA-7617 URL: https://issues.apache.org/jira/browse/KAFKA-7617 Project: Kafka Issue Type: Task Reporter: Viktor Somogyi Assignee: Viktor Somogyi Although the documentation gives help on configuring the authentication and authorization, it won't list what are the security primitives (operations and resources) that can be used which makes it hard for users to easily set up thorough authorization rules. This task would cover adding these to the security page of the Kafka documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7433) Introduce broker options in TopicCommand to use AdminClient
Viktor Somogyi created KAFKA-7433: - Summary: Introduce broker options in TopicCommand to use AdminClient Key: KAFKA-7433 URL: https://issues.apache.org/jira/browse/KAFKA-7433 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 2.1.0 Reporter: Viktor Somogyi Assignee: Viktor Somogyi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7140) Remove deprecated poll usages
Viktor Somogyi created KAFKA-7140: - Summary: Remove deprecated poll usages Key: KAFKA-7140 URL: https://issues.apache.org/jira/browse/KAFKA-7140 Project: Kafka Issue Type: Improvement Reporter: Viktor Somogyi Assignee: Viktor Somogyi There are a couple of poll(long) usages of the consumer in test and non-test code. This jira would aim to remove the non-test usages of the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7087) Option in TopicCommand to report not preferred leaders
Viktor Somogyi created KAFKA-7087: - Summary: Option in TopicCommand to report not preferred leaders Key: KAFKA-7087 URL: https://issues.apache.org/jira/browse/KAFKA-7087 Project: Kafka Issue Type: Improvement Components: core Reporter: Viktor Somogyi Assignee: Viktor Somogyi Options in topic describe exists for reporting unavailable and lagging partitions but it is often an ask to report partitions where the active leader is not the preferred one. This jira adds this extra option to TopicCommand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6187) Remove the Logging class in favor of LazyLogging in scala-logging
Viktor Somogyi created KAFKA-6187: - Summary: Remove the Logging class in favor of LazyLogging in scala-logging Key: KAFKA-6187 URL: https://issues.apache.org/jira/browse/KAFKA-6187 Project: Kafka Issue Type: Task Components: core Reporter: Viktor Somogyi Assignee: Viktor Somogyi In KAFKA-1044 we removed the hard dependency on junit and enabled users to exclude it in their environment without causing any problems. We also agreed to remove the kafka.utils.Logging class as it can be made redundant by LazyLogging in scala-logging. In this JIRA we will get rid of Logging by replacing its remaining functionalities with other features. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6084) ReassignPartitionsCommand should propagate JSON parsing failures
Viktor Somogyi created KAFKA-6084:
-
Summary: ReassignPartitionsCommand should propagate JSON parsing
failures
Key: KAFKA-6084
URL: https://issues.apache.org/jira/browse/KAFKA-6084
Project: Kafka
Issue Type: Improvement
Components: admin
Affects Versions: 0.11.0.0
Reporter: Viktor Somogyi
Assignee: Viktor Somogyi
Priority: Minor
Attachments: Screen Shot 2017-10-18 at 23.31.22.png
Basically looking at Json.scala it will always swallow any parsing errors:
{code}
def parseFull(input: String): Option[JsonValue] =
try Option(mapper.readTree(input)).map(JsonValue(_))
catch { case _: JsonProcessingException => None }
{code}
However sometimes it is easy to figure out the problem by simply looking at the
JSON, in some cases it is not very trivial, such as some invisible characters
(like byte order mark) won't be displayed by most of the text editors and can
people spend time on figuring out what's the problem.
As Jackson provides a really detailed exception about what failed and how, it
is easy to propagate the failure to the user.
As an example I attached a BOM prefixed JSON which fails with the following
error which is very counterintuitive:
{noformat}
[root@localhost ~]# kafka-reassign-partitions --zookeeper localhost:2181
--reassignment-json-file /root/increase-replication-factor.json --execute
Partitions reassignment failed due to Partition reassignment data file
/root/increase-replication-factor.json is empty
kafka.common.AdminCommandFailedException: Partition reassignment data file
/root/increase-replication-factor.json is empty
at
kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:120)
at
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:52)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
...
{noformat}
In case of the above error it would be much better to see what fails exactly:
{noformat}
kafka.common.AdminCommandFailedException: Admin command failed
at
kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:267)
at
kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:275)
at
kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:197)
at
kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:193)
at
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:64)
at
kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character
('' (code 65279 / 0xfeff)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: (String)"{"version":1,
"partitions":[
{"topic": "test1", "partition": 0, "replicas": [1,2]},
{"topic": "test2", "partition": 1, "replicas": [2,3]}
]}"; line: 1, column: 2]
at
com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1798)
at
com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:663)
at
com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:561)
at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1892)
at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:747)
at
com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4030)
at
com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2539)
at kafka.utils.Json$.kafka$utils$Json$$doParseFull(Json.scala:46)
at kafka.utils.Json$$anonfun$tryParseFull$1.apply(Json.scala:44)
at kafka.utils.Json$$anonfun$tryParseFull$1.apply(Json.scala:44)
at scala.util.Try$.apply(Try.scala:192)
at kafka.utils.Json$.tryParseFull(Json.scala:44)
at
kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:241)
... 5 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5723) Refactor BrokerApiVersionsCommand to use the KafkaAdminClient
Viktor Somogyi created KAFKA-5723: - Summary: Refactor BrokerApiVersionsCommand to use the KafkaAdminClient Key: KAFKA-5723 URL: https://issues.apache.org/jira/browse/KAFKA-5723 Project: Kafka Issue Type: Improvement Reporter: Viktor Somogyi Assignee: Viktor Somogyi Currently it uses the deprecated AdminClient and in order to remove usages of that client, this class needs to be refactored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5722) Refactor ConfigCommand to use the new AdminClient
Viktor Somogyi created KAFKA-5722: - Summary: Refactor ConfigCommand to use the new AdminClient Key: KAFKA-5722 URL: https://issues.apache.org/jira/browse/KAFKA-5722 Project: Kafka Issue Type: Improvement Reporter: Viktor Somogyi Assignee: Viktor Somogyi The ConfigCommand currently uses a direct connection to zookeeper. The zookeeper dependency should be deprecated and an AdminClient API created to be used instead. This change requires a KIP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-1044) change log4j to slf4j
[ https://issues.apache.org/jira/browse/KAFKA-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044842#comment-16044842 ] Viktor Somogyi edited comment on KAFKA-1044 at 6/9/17 6:53 PM: --- [~ewencp], thank you, started working on it. I found two incompatibilities and I'm asking your advice in the resolution. * Fatal logging level: slf4j doesn't support it * kafka.utils.Log4jController: there are some log4j specific features used here that can't be replaced with a generic solution I have two solutions for these: # we could simply use the log4j-over-slf4j package. This will redirect all Fatal logs to Error level and also provides some of the functionalities required by Log4jController (but not all). * Pros: a few lines of changes only, quite straightforward * Cons: loss of features. However Fatal can't be worked around so we have to accept that, but we also lose Log4jController.getLoggers as there is no way of collecting the current loggers by simply relying on slf4j. Also other methods' behaviour will change (no way to detect existing loggers in slf4j as far as I'm aware of it) # we could separate off Log4jController into a separate module and have the users to put it on the classpath explicitly if they use log4j for logging. From Logging class we can instantiate it by reflection. * Pros: except Fatal logging level we keep all the features * Cons: more complicated and breaking, also requires documentation so users will be aware of this I'll try to create PRs of both solutions (and they are rather work in progress, I just want to show approximate solutions) just in case if that is fine. was (Author: viktorsomogyi): [~ewencp], thank you, started working on it. I found two incompatibilities and I'm asking your advice in the resolution. * Fatal logging level: slf4j doesn't support it * kafka.utils.Log4jController: there are some log4j specific features used here that can't be replaced with a generic solution I have two solutions for these: # we could simply use the log4j-over-slf4j package. This will redirect all Fatal logs to Error level and also provides some of the functionalities required by Log4jController (but not all). * Pros: a few lines of changes only, quite straightforward * Cons: loss of features. However Fatal can't be worked around so we have to accept that, but we also lose Log4jController.getLoggers as there is no way of collecting the current loggers by simply relying on slf4j. Also other methods' behaviour will change (no way to detect existing loggers in slf4j as far as I'm aware of it) # we could separate off Log4jController into a separate module and have the users to put it on the classpath explicitly if they use log4j for logging. From Logging class we can instantiate it by reflection. * Pros: except Fatal logging level we keep all the features * Cons: more complicated and breaking, also requires documentation so users will be aware of this I'm attaching patches of both solutions (and they are rather work in progress, I just want to show approximate solutions) just in case > change log4j to slf4j > -- > > Key: KAFKA-1044 > URL: https://issues.apache.org/jira/browse/KAFKA-1044 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.0 >Reporter: shijinkui >Assignee: Viktor Somogyi >Priority: Minor > Labels: newbie > > can u chanage the log4j to slf4j, in my project, i use logback, it's conflict > with log4j. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1044) change log4j to slf4j
[ https://issues.apache.org/jira/browse/KAFKA-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044842#comment-16044842 ] Viktor Somogyi commented on KAFKA-1044: --- [~ewencp], thank you, started working on it. I found two incompatibilities and I'm asking your advice in the resolution. * Fatal logging level: slf4j doesn't support it * kafka.utils.Log4jController: there are some log4j specific features used here that can't be replaced with a generic solution I have two solutions for these: # we could simply use the log4j-over-slf4j package. This will redirect all Fatal logs to Error level and also provides some of the functionalities required by Log4jController (but not all). * Pros: a few lines of changes only, quite straightforward * Cons: loss of features. However Fatal can't be worked around so we have to accept that, but we also lose Log4jController.getLoggers as there is no way of collecting the current loggers by simply relying on slf4j. Also other methods' behaviour will change (no way to detect existing loggers in slf4j as far as I'm aware of it) # we could separate off Log4jController into a separate module and have the users to put it on the classpath explicitly if they use log4j for logging. From Logging class we can instantiate it by reflection. * Pros: except Fatal logging level we keep all the features * Cons: more complicated and breaking, also requires documentation so users will be aware of this I'm attaching patches of both solutions (and they are rather work in progress, I just want to show approximate solutions) just in case > change log4j to slf4j > -- > > Key: KAFKA-1044 > URL: https://issues.apache.org/jira/browse/KAFKA-1044 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.0 >Reporter: shijinkui >Assignee: Viktor Somogyi >Priority: Minor > Labels: newbie > > can u chanage the log4j to slf4j, in my project, i use logback, it's conflict > with log4j. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Issue Comment Deleted] (KAFKA-1044) change log4j to slf4j
[ https://issues.apache.org/jira/browse/KAFKA-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi updated KAFKA-1044: -- Comment: was deleted (was: @ewencp, thank you for your help. I've started this task and all in all my conclusion is that we should delay this until the release of slf4j 1.8 (alpha2 is already out), as there are some log4j features that are being used but not part of slf4j, such as the LogManager in Log4jController or the Fatal log level. It is a bit hard to work around these effectively and without breaking behaviour. Best I know is we could use error level instead of fatal for the time being or we could separate off Log4jController into its own module and try to load it in runtime if log4j is being used to get rid of the hard log4j dependency, but these are quite hacky and I'm not sure if it's worth the effort now as we have a workaround for the original issue. Please let me know what you think.) > change log4j to slf4j > -- > > Key: KAFKA-1044 > URL: https://issues.apache.org/jira/browse/KAFKA-1044 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.0 >Reporter: shijinkui >Assignee: Viktor Somogyi >Priority: Minor > Labels: newbie > > can u chanage the log4j to slf4j, in my project, i use logback, it's conflict > with log4j. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1044) change log4j to slf4j
[ https://issues.apache.org/jira/browse/KAFKA-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043329#comment-16043329 ] Viktor Somogyi commented on KAFKA-1044: --- @ewencp, thank you for your help. I've started this task and all in all my conclusion is that we should delay this until the release of slf4j 1.8 (alpha2 is already out), as there are some log4j features that are being used but not part of slf4j, such as the LogManager in Log4jController or the Fatal log level. It is a bit hard to work around these effectively and without breaking behaviour. Best I know is we could use error level instead of fatal for the time being or we could separate off Log4jController into its own module and try to load it in runtime if log4j is being used to get rid of the hard log4j dependency, but these are quite hacky and I'm not sure if it's worth the effort now as we have a workaround for the original issue. Please let me know what you think. > change log4j to slf4j > -- > > Key: KAFKA-1044 > URL: https://issues.apache.org/jira/browse/KAFKA-1044 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.0 >Reporter: shijinkui >Assignee: Viktor Somogyi >Priority: Minor > Labels: newbie > > can u chanage the log4j to slf4j, in my project, i use logback, it's conflict > with log4j. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1044) change log4j to slf4j
[ https://issues.apache.org/jira/browse/KAFKA-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039920#comment-16039920 ] Viktor Somogyi commented on KAFKA-1044: --- Hi, I'm new to Kafka and was searching for a fitting task. If this is still relevant then I'd like to pick it up. Could someone please add me as contributor so I could assign it to me? > change log4j to slf4j > -- > > Key: KAFKA-1044 > URL: https://issues.apache.org/jira/browse/KAFKA-1044 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.0 >Reporter: shijinkui >Priority: Minor > Labels: newbie > > can u chanage the log4j to slf4j, in my project, i use logback, it's conflict > with log4j. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
