Re: Use one producer for both coordinator stream and users system?
Hi Yan, Naive question: what do we need producer thread of coordinator stream for? Thanks, -Tao On Mon, Aug 17, 2015 at 2:09 PM, Yan Fang yanfang...@gmail.com wrote: Hi guys, I have this question because Kafka's doc http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html seems recommending having one producer shared by all threads (*The producer is thread safe and should generally be shared among all threads for best performance.*), while currently the coordinator stream is using a separate producer (usually, there are two producers(two producer threads) in each container: one is for the coordinator stream , one is for the real job) 1. Will having one producer shared by all thread really improve the performance? (haven't done the perf test myself. Guess Kafka has some proof). 2. if yes, should we go this way? Thanks, Fang, Yan yanfang...@gmail.com
Review Request 37528: SAMZA-736 BrokerProxy will stuck in infinite loop if consumer.fetch throws OOME
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37528/ --- Review request for samza. Repository: samza Description --- Added new catch blocks to prevent infinite loops Diffs - samza-core/src/main/scala/org/apache/samza/util/ExponentialSleepStrategy.scala 376b277 samza-kafka/src/main/scala/org/apache/samza/system/kafka/BrokerProxy.scala 614f33f Diff: https://reviews.apache.org/r/37528/diff/ Testing --- Thanks, Aleksandar Pejakovic
Re: [Discuss/Vote] upgrade to Yarn 2.6.0
Hi, Yan, Thanks for rolling the ball! +1 from me to upgrade the minimum supported version to YARN 2.6, assuming that we are going to fix SAMZA-750 together. On Mon, Aug 17, 2015 at 4:41 PM, Yan Fang yanfang...@gmail.com wrote: Hi guys, we have been discussing upgrading to Yarn 2.6.0 (SAMZA-536 https://issues.apache.org/jira/browse/SAMZA-536), because there are some bug fixes after 2.4.0 and we can not enable the Yarn RM recovering feature in Yarn 2.4.0 (SAMZA-750 https://issues.apache.org/jira/browse/SAMZA-750 ) . So we just want to make sure if any production users are still using Yarn 2.4.0 and do not plan to upgrade to 2.6.0+? If not further concern, I think we can go and upgrade to Yarn 2.6.0 in Samza 0.10.0 release. Thanks, Fang, Yan yanfang...@gmail.com
Re: Review Request 37536: SAMZA-710 Update WebServlet and RestServlet to read coordinatorStream information
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37536/#review95672 --- I believe this patch works, but could you also add a unit test to verify the correctness? Also overusing the localityManager is a little concerned, could you think of another way? Thank you. samza-core/src/main/java/org/apache/samza/job/model/JobModel.java (line 74) https://reviews.apache.org/r/37536/#comment150767 this is a little overuse of the localityManager. It is not designed to get the latest config. samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterRestServlet.scala (line 106) https://reviews.apache.org/r/37536/#comment150765 sanitize it as well. - Yan Fang On Aug. 17, 2015, 2:56 p.m., Aleksandar Bircakovic wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37536/ --- (Updated Aug. 17, 2015, 2:56 p.m.) Review request for samza. Repository: samza Description --- WebServlet and RestServlet now read information from coordinator stream consumer and get new config. Diffs - samza-core/src/main/java/org/apache/samza/coordinator/stream/AbstractCoordinatorStreamManager.java ca97ce8 samza-core/src/main/java/org/apache/samza/job/model/JobModel.java 7b59274 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterRestServlet.scala 09f4dc3 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterWebServlet.scala 7fd5122 Diff: https://reviews.apache.org/r/37536/diff/ Testing --- Thanks, Aleksandar Bircakovic
[Discuss/Vote] upgrade to Yarn 2.6.0
Hi guys, we have been discussing upgrading to Yarn 2.6.0 (SAMZA-536 https://issues.apache.org/jira/browse/SAMZA-536), because there are some bug fixes after 2.4.0 and we can not enable the Yarn RM recovering feature in Yarn 2.4.0 (SAMZA-750 https://issues.apache.org/jira/browse/SAMZA-750) . So we just want to make sure if any production users are still using Yarn 2.4.0 and do not plan to upgrade to 2.6.0+? If not further concern, I think we can go and upgrade to Yarn 2.6.0 in Samza 0.10.0 release. Thanks, Fang, Yan yanfang...@gmail.com
Re: remote kafka producer -- kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries
Hi, All:
Finally I fixed this bug.
1. set advertised.host.name at config/server.properites as AWS
*private IP* address
(not public *DNS*)
2. comment host.name at config/server.properites
3. In remote java producer:
props.put(metadata.broker.list, borkerPrivateIp+ :9092);
The value brokerPrivateIp is same as advertised.host.name
at config/server.properites
This bug blocked me a while...
Sincerely,
Selina
On Thu, Aug 13, 2015 at 9:56 PM, Job-Selina Wu swucaree...@gmail.com
wrote:
Dear All:
I got kafka.common.FailedToSendMessageException: Failed to send
messages after 3 tries as below. When I have a remote java Kafka producer
try to produce message to Kafka broker Server. Both Producer and Broker are
at AWS cloud. BTW, I tried my code first at local machine and Virtual
machine first, It did not work either.(advertised.host.name was set to
the ip address of the kafka server*)*
*-**This is my KafkaProducer at remote Producer for producer
configuration**-*
public class KafkaProducer {
Properties props = new Properties();
private final ProducerString, String producer;
private final String kafkaServerIP = 52.19.2.74:9092;
public KafkaProducer() {
props.put(metadata.broker.list, kafkaServerIP);
//props.put(bootstrap.servers, localhost:9092 );
props.put(serializer.class, kafka.serializer.StringEncoder);
props.put(advertised.host.name, localhost);
props.put(request.required.acks, 0);
ProducerConfig config = new ProducerConfig(props);
producer = new ProducerString, String(config);
}
public ProducerString, String getProducer() {
return this.producer;
}
}
*The configs/server.properties at Kafka Server at AWS*-
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
broker.id=0
port=9092
host.name=localhost
*advertised.host.name http://advertised.host.name*=
ec2-51-18-21-235.us-west-1.compute.amazonaws.com
# below is same as default
#advertised.port=port accessible by clients
#advertised.port=port accessible by clients
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
#log.flush.interval.messages=1
#log.flush.interval.ms=1000
log.retention.hours=168
#log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=30
log.cleaner.enable=false
- - --- - - - - --
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:77)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at com.cinarra.kafka.Main.main(Main.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
reference:
http://stackoverflow.com/questions/30217255/cant-connect-to-a-remote-kafka-producer-from-windows-through-java-code
Your help is highly appreciated,
Selina
Review Request 37536: SAMZA-710 Update WebServlet and RestServlet to read coordinatorStream information
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37536/ --- Review request for samza. Repository: samza Description --- WebServlet and RestServlet now read information from coordinator stream consumer and get new config. Diffs - samza-core/src/main/java/org/apache/samza/coordinator/stream/AbstractCoordinatorStreamManager.java ca97ce8 samza-core/src/main/java/org/apache/samza/job/model/JobModel.java 7b59274 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterRestServlet.scala 09f4dc3 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterWebServlet.scala 7fd5122 Diff: https://reviews.apache.org/r/37536/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: remote kafka producer -- kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries
Hi, Yan:
Thanks for your reply. You are right. I just subscribed the mail list
for kafka us...@kafka.apache.org
Sincerely,
Selina
On Sun, Aug 16, 2015 at 4:04 PM, Yan Fang yanfang...@gmail.com wrote:
Hi Selina,
I guess you can post this question in the Kafka mailing list if this is
pure kafka, there are more experts in that community, though there are
some Kafka experts here as well. :)
Thanks,
Fang, Yan
yanfang...@gmail.com
On Thu, Aug 13, 2015 at 9:56 PM, Job-Selina Wu swucaree...@gmail.com
wrote:
Dear All:
I got kafka.common.FailedToSendMessageException: Failed to send
messages after 3 tries as below. When I have a remote java Kafka producer
try to produce message to Kafka broker Server. Both Producer and Broker are
at AWS cloud. BTW, I tried my code first at local machine and Virtual
machine first, It did not work either.(advertised.host.name was set to
the ip address of the kafka server*)*
*-**This is my KafkaProducer at remote Producer for producer
configuration**-*
public class KafkaProducer {
Properties props = new Properties();
private final ProducerString, String producer;
private final String kafkaServerIP = 52.19.2.74:9092;
public KafkaProducer() {
props.put(metadata.broker.list, kafkaServerIP);
//props.put(bootstrap.servers, localhost:9092 );
props.put(serializer.class, kafka.serializer.StringEncoder);
props.put(advertised.host.name, localhost);
props.put(request.required.acks, 0);
ProducerConfig config = new ProducerConfig(props);
producer = new ProducerString, String(config);
}
public ProducerString, String getProducer() {
return this.producer;
}
}
*The configs/server.properties at Kafka Server at AWS*-
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
broker.id=0
port=9092
host.name=localhost
*advertised.host.name http://advertised.host.name*=
ec2-51-18-21-235.us-west-1.compute.amazonaws.com
# below is same as default
#advertised.port=port accessible by clients
#advertised.port=port accessible by clients
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
#log.flush.interval.messages=1
#log.flush.interval.ms=1000
log.retention.hours=168
#log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=30
log.cleaner.enable=false
- - --- - - - - --
kafka.common.FailedToSendMessageException: Failed to send messages after
3 tries.
kafka.common.FailedToSendMessageException: Failed to send messages after
3 tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:77)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at com.cinarra.kafka.Main.main(Main.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
reference:
http://stackoverflow.com/questions/30217255/cant-connect-to-a-remote-kafka-producer-from-windows-through-java-code
Your help is highly appreciated,
Selina
Use one producer for both coordinator stream and users system?
Hi guys, I have this question because Kafka's doc http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html seems recommending having one producer shared by all threads (*The producer is thread safe and should generally be shared among all threads for best performance.*), while currently the coordinator stream is using a separate producer (usually, there are two producers(two producer threads) in each container: one is for the coordinator stream , one is for the real job) 1. Will having one producer shared by all thread really improve the performance? (haven't done the perf test myself. Guess Kafka has some proof). 2. if yes, should we go this way? Thanks, Fang, Yan yanfang...@gmail.com
Review Request 37521: SAMZA-737 property name stores.store-name.changelog.topic-level-property is not effective
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37521/ --- Review request for samza. Repository: samza Description --- For issue described in [SAMZA-737](https://issues.apache.org/jira/browse/SAMZA-737) changed property stores.store-name.changelog.topic-level-property to stores.store-name.changelog.kafka.topic-level-property Diffs - docs/learn/documentation/versioned/jobs/configuration-table.html 8177fe5 Diff: https://reviews.apache.org/r/37521/diff/ Testing --- Thanks, Aleksandar Bircakovic
