where you delegate the token validation to the authorization
server.
[1] https://github.com/strimzi/strimzi-kafka-oauth
- marko
On 2020/04/08 04:37:09, Antony Alphonse wrote:
> Hi,
>
> I'm looking to implement authentication using Oauthbearer mechanism in my
> Kafka cluster. My
] https://github.com/strimzi/strimzi-kafka-oauth
- marko
On 2020/04/08 04:37:09, Antony Alphonse wrote:
> Hi,
>
> I'm looking to implement authentication using Oauthbearer mechanism in my
> Kafka cluster. My Oauth server will be Azure AD. If anyone have implemented
> similar setup o
This would be useful, for example, showing messages belonging to the
same transaction.
thanks
marko
- Original Message -
From: users@kafka.apache.org
To:
Cc:
Sent:Tue, 31 Jul 2018 14:40:20 -0700
Subject:Re: Viewing transactional markers in client
Is there any way for a KafkaConsumer to view/get the transactional
marker messages?
--
Best regards,
Marko
www.kafkatool.com
re successfully written to disk even in case of power failure,
>>provided disks didn't crash ?
No, see my above reply.
-Dave
-Original Message-
From: JEVTIC, MARKO [mailto:marko.jev...@fisglobal.com]
Sent: Tuesday, May 30, 2017 8:05 AM
To: users@kafka.apache.org
Subject: client recor
You can use something like this to get a comma-separated list of all filed
in a folder:
ls -l | awk '{print $9}' ORS=','
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http:/
Do you know in advance (when sending the first message) how many messages
that batch is going to have?
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://semat
nc replica becomes
the leader.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Thu, Sep 29, 2016 at 7:30 PM, Ezra Stuetzel <e
enough for any type of use case? What do
you think cannot be achieved?
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Thu, Sep 15,
BTW regarding latency:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Con
lag; i.e. lag can fit
in the OS page cache so you're not even hitting disk when consuming)
measured in low 10s of ms.
No read replicas. You only read from partition's master. I.e. replicas are
used to achieve redundancy.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log
Hi Karin,
regarding 5 (fsyncing to disk), take a look at the broker configuration
parameters whose names start with log.flush.
http://kafka.apache.org/documentation.html#brokerconfigs
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Sup
Hi Tom,
if you need a commercially proven lag monitoring solution (and all other
Kafka and ZK metrics) take a look at our SPM.
Hope you don't mind me plugging this one in :)
[image: Inline image 1]
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr
Is there a way to get a list of all consumer groups and consumer group
offsets using either KafkaConsumer or KafkaProducer (or some other method)
in the new Java client?
Best regards,
Marko
www.kafkatool.com
to assign the second consumer with a different group, each
consumer would consume all messages (independently of one another).
BTW, Kafka is not broadcasting anything, your consumers are pulling
messages out of Kafka :)
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log
Instantly reminded me of Streams API, where you can use Java8 streams
semantics (filter being one of them) to do the first thing in Gouzhang's
response (filter messages from one topic into another - I assume that's
what you were looking for).
Marko Bonaći
Monitoring | Alerting | Anomaly Detection
Ah, that makes sense. After adding the truststore to the server configs
things seem to work correctly, thanks!
marko
> Have you configured a truststore in server.properties? You don't need this
> when using security.inter.broker.protocol=PLAINTEXT and client-auth is
> disabled, but yo
], Error during controlled
shutdown, possibly because leader movement took longer than the configured
socket.timeout.ms: Connection to Node(0, debian, 9094) failed
(kafka.server.KafkaServer)
the relevant configs are
listeners=SSL://:9094
security.inter.broker.protocol=SSL
port=9094
Marko
> If y
I'm assuming that you created a topic with replication factor 3, while
having only a single broker.
Try with replication factor 1 or add additional brokers.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext &
There is only one broker in this case. There are no errors (besides the
warning below) on either the broker or the client side. It just returns an
empty topic list if plaintext is not configured, even though client is
using SSL in both cases.
marko
> Hi,
>
> That warning is harmless. P
by the
consumer in both cases, I'm not sure why having the plaintext port would
affect the SSL behavior.
--
Best regards,
Marko
www.kafkatool.com
Thanks, this seems to do the trick.
Best regards,
Marko
www.kafkatool.com
> Hi
>
> For the Kafka Consumer, there are seekToBeginning and seekToEnd method
> that
> point your the beginning and end of the partition. You can use one of the
> methods to point the consumer to
How does one get the first/last offset for a given partition using the new
KafkaConsumer/Producer? In the old SimpleConsumer you would just use the
getOffsetsBefore() method.
--
Best regards,
Marko
www.kafkatool.com
Also sent to: ggol...@hortonworks.com
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Fri, Apr 15, 2016 at 1:47 AM, Gurudi
Automated reply:
thank you for attempting to subscribe to Kafka mailing list.
To finish the subscription process send email to
users-subscr...@kafka.apache.org < users-subscr...@kafka.apache.org>
:)
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Managemen
+
Has anything changed in 0.9?
Thanks
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
-providers
Amazon Kinesis would also work.
Anything really that would "outsource" the initial effort until you're
ready to commit to Kafka.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sem
regardless of the load.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Mon, Mar 21, 2016 at 6:25 PM, Ben Stopford <b...@
These two issues track progress of Kafka consumer 0.9.
https://github.com/apache/spark/pull/10953
https://github.com/apache/spark/pull/11143
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/>
Is there a way to detect the broker version (even at a high level 0.8 vs
0.9) using the kafka-clients Java library?
--
Best regards,
Marko
www.kafkatool.com
Does the new KafkaConsumer support storing offsets in Zookeeper or only in
Kafka? By looking at the source code I could not find any support for
Zookeeper, but wanted to confirm this.
--
Best regards,
Marko
www.kafkatool.com
Hello again Cosmin :)
I think this is because offsets are kept in a special _consumer_offsets
topic, as opposed to ZK previously.
Take a look here:
http://search-hadoop.com/m/uyzND1T1i3BNkRFM1=Re+Kafka+0+8+2+1+how+to+read+from+__consumer_offsets+topic+
Marko Bonaći
Monitoring | Alerting | Anomaly
Hi Cosmin,
do you have default server configuration on these new nodes you're setting
up?
I'd check consumer's socket.timeout.ms, maybe someone set it to 30 instead
of 30 000 :)
Speaking from my own experience (I had the same symptom and this turned out
to be the cause).
Marko Bonaći
Monitoring
Actually, why don't you use the same code as outlined here (that includes
timeout in props):
http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Sup
n requests?
FINALLY: tell us more about your use case.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Mon, Jan 4, 2016 at 1
here:
https://github.com/quantifind/KafkaOffsetMonitor
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Wed, Dec 30, 2015 at
by default)
I'm still on Kafka 0.8, so I can't shed any light on your issue.
Thx for the AdminClient info.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about
I was refering to Dana Powers's answer in the link I posted (to use a
client API). You can find an example here:
http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr
We recently had the same question:
http://search-hadoop.com/m/uyzND1kM7q1gElhy1
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On T
-class.sh kafka.tools.ImportZkOffsets --input-file
/tmp/zk-offsets --zkconnect localhost:2181
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
Hmm, I guess you're right Tod :)
Just to confirm, you meant that, while you're changing the exported file it
might happen that one of the segment files becomes eligible for cleanup by
retention, which would then make the imported offsets out of range?
Marko Bonaći
Monitoring | Alerting | Anomaly
Spotify's image since it contains both ZK and Kafka, but I think
the latest version they built is 0.8.2.1, so you might have to build the
new image yourself if you need 0.9, but that's trivial to do.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr
Hi,
there was a problem with JMX consumer lag in 0.8:
http://search-hadoop.com/m/uyzND14v72215XZpK=Re+Consumer+lag+lies+orphaned+offsets+
Has anything changed now with 0.9?
Thanks
Did not know that quotas landed in 0.9. Very nice!
Being able to throttle clients that don't have real-time SLAs (in favor of
those who do) is a great addition.
Thanks for that Grant.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Sup
case.
Perhaps you could check Consumer offsets from your Producer and then decide
based on that information whether to throttle Producer or not. Could get
complicated really fast, though.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Sup
AFAIK there is no such notion as maximum length of a topic, i.e. offset has
no limit, except Long.MAX_VALUE I think, which should be enough for a
couple of lifetimes (9 * 10E18, or quintillion or million trillions).
What would be the purpose of that, besides being a nice foot-gun :)
Marko Bonaći
running producers, brokers and consumers on AWS.
Is it possible that the network is that much flaky?
What's your experience?
Thanks,
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Con
g the same thing?
Or should I just forget about it and use the recommended approach from the
low-level consumer code example in the wiki (which I currently use as the
fallback)?
Thanks,
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Su
consumers, including Apache Storm Kafka spout
consumers
-Show JSON and XML messages in a pretty-printed format
-Export messages from a topic to files on local hard drive
Feedback and comments are welcome, you can find the tool at www.kafkatool.com
Cheers,
Marko
49 matches
Mail list logo
