Yi,
Thanks a lot for all the help. I am able to get the sliding window working.
Here is what I have now to get it to work:
for (KeyValueIterator i = store.all(); i.hasNext();) {
Entry next= (Entry) i.next();
String key = (String) next.getKey();
String value = (String) s
Yeah, that's why I added some test code in the window() to call store.all()
and iterate through. I traced into it in my local environment and verified
that the iterator is functioning with store.all().
-Yi
On Thu, Jul 23, 2015 at 4:26 PM, Shekar Tippur wrote:
> Yi,
>
> In my case, I am able to
Yi,
In my case, I am able to append to the key but I am not able to get the
store and iterate through.
If you look at http://pastebin.com/fKGpHwW6, line 146, I am able to get the
store value. but in window routine - line 187, I am unable to get the
values from store.
- Shekar
Hi, Shekar,
I was merely testing whether the counter per key works, if that makes sense
to your use case.
-Yi
On Thu, Jul 23, 2015 at 3:25 PM, Shekar Tippur wrote:
> Yi,
>
> I am new to Scala. While it is readable, I am not sure where you are
> incrementing the count per application?
>
> - She
Yi,
I am new to Scala. While it is readable, I am not sure where you are
incrementing the count per application?
- Shekar
On Wed, Jul 22, 2015 at 5:20 PM, Shekar Tippur wrote:
> Thanks Yi. I got the pastebin link.
> I am looking at it.
>
> Shekar
> On Jul 22, 2015 5:09 PM, "Yi Pan" wrote:
>
>
Thanks Yi. I got the pastebin link.
I am looking at it.
Shekar
On Jul 22, 2015 5:09 PM, "Yi Pan" wrote:
> Hmm... did you get my pastebin post? If not, I can send the diff directly
> to your gmail account.
>
> On Wed, Jul 22, 2015 at 11:23 AM, Shekar Tippur wrote:
>
> > Yi,
> >
> > I am not sure
Hmm... did you get my pastebin post? If not, I can send the diff directly
to your gmail account.
On Wed, Jul 22, 2015 at 11:23 AM, Shekar Tippur wrote:
> Yi,
>
> I am not sure if attachments gets filtered when sent to the group. I have
> seen this earlier as well.
>
> - Shekar
>
Hi, Shekar,
Here it is: http://pastebin.com/fKGpHwW6
-Yi
On Wed, Jul 22, 2015 at 8:05 AM, Shekar Tippur wrote:
> Yi,
>
> I am not sure I see an attachment. Is it possible to paste that on
> pastebin?
>
> Shekar
> On Jul 21, 2015 4:27 PM, "Yi Pan" wrote:
>
> > Hi, Shekar,
> >
> > I have stri
Yi,
I am not sure if attachments gets filtered when sent to the group. I have
seen this earlier as well.
- Shekar
Yi,
I am not sure I see an attachment. Is it possible to paste that on
pastebin?
Shekar
On Jul 21, 2015 4:27 PM, "Yi Pan" wrote:
> Hi, Shekar,
>
> I have strip down your use case just to the KV-store operation and have
> verified that it works fine. Please see the attached diff file.
>
> If yo
Hi, Shekar,
I have strip down your use case just to the KV-store operation and have
verified that it works fine. Please see the attached diff file.
If you have any further questions, please let me know.
-Yi
On Mon, Jul 20, 2015 at 12:35 PM, Shekar Tippur wrote:
> Yi,
>
> Here is the config:
>
Yi,
Here is the config:
http://pastebin.com/mCALEACs
- Shekar
On Mon, Jul 20, 2015 at 12:27 PM, Yi Pan wrote:
> Hi, Shekar,
>
> It would also be helpful if you can post your job configuration on the
> pastebin s.t. I can test the same config.
>
> Thanks!
>
> -Yi
>
> On Mon, Jul 20, 2015 at 11:
Hi, Shekar,
It would also be helpful if you can post your job configuration on the
pastebin s.t. I can test the same config.
Thanks!
-Yi
On Mon, Jul 20, 2015 at 11:11 AM, Shekar Tippur wrote:
> Yi,
>
> Thanks a lot.
>
> - Shekar
>
Yi,
Thanks a lot.
- Shekar
Hi, Shekar,
I have copied over your code and made some simplification to focus on the
KV-store operation. I have not fully integrated w/ the system to start the
test yet. I will follow up tomorrow.
-Yi
On Fri, Jul 17, 2015 at 2:02 PM, Shekar Tippur wrote:
> Yi,
>
> Here is the code:
>
> http:/
Yi,
Here is the code:
http://pastebin.com/y5iSLcuK
(Pls note that this has some redundant code as I am playing with it.)
- Shekar
On Fri, Jul 17, 2015 at 1:47 PM, Yi Pan wrote:
> Hi, Shekar,
>
> If possible, could you share your code somewhere? I can try to dig into it
> this weekend.
>
> Th
Hi, Shekar,
If possible, could you share your code somewhere? I can try to dig into it
this weekend.
Thanks!
-Yi
On Fri, Jul 17, 2015 at 1:31 PM, Shekar Tippur wrote:
> Any takers on this please?
>
> - Shekar
>
Any takers on this please?
- Shekar
Yi,
Interesting;y, I see these in changelog topic (Guessing these are window
counts):
kafka-console-consumer.sh --zookeeper localhost:2181 --topic
window-changelog
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SL
Yi,
I see incoming messages. I see that the counts are getting aggregated as
well.
But when I try to access it, I get null.
- Shekar
On Mon, Jul 6, 2015 at 4:50 PM, Yi Pan wrote:
> Hi, Shekar,
>
> Did you take a look at the stats to see:
>
> 1) Is there any incoming messages?
> 2) Is there any
Hi, Shekar,
Did you take a look at the stats to see:
1) Is there any incoming messages?
2) Is there any messages in the changelog topic?
Could you also try to change the log4j level to DEBUG to see whether we can
see something in the log?
Thanks!
-Yi
On Mon, Jul 6, 2015 at 4:43 PM, Shekar Tip
Martin,
As seen below, I have only 1 partition. What else could be wrong?
*$ *kafka-topics.sh --describe --zookeeper localhost:2181 --topic parser
Topic:parser PartitionCount:1 ReplicationFactor:1 Configs:
Topic: parser Partition: 0 Leader: 0 Replicas: 0 Isr: 0
- Shekar
On Mon, Jul 6, 2015
Hi Shekar,
The store.all() iterator ought to give you the entire contents of the store.
However, note that each partition of the input topic results in a separate
StreamTask instance, which in turn has a separate store. So there will be as
many stores as there are input partitions. Perhaps you'
Any takers on this please?
- Shekar
On Fri, Jul 3, 2015 at 9:46 AM, Shekar Tippur wrote:
> Any answer on how to get all the kv values and reinitialise the kv store?
>
> Had one more question on implementing sliding window.
>
> If i use a kv store like rocksdb, and I use yarn (say 3 node cluster
Any answer on how to get all the kv values and reinitialise the kv store?
Had one more question on implementing sliding window.
If i use a kv store like rocksdb, and I use yarn (say 3 node cluster), the
job that it runs to aggregate gets distributed as well and I am guessing
the aggregation numbe
Also, next.getValue() or next.getKey() does not yield anything.
KeyValueIterator i = store.all();
while(i.hasNext()){
Entry next = i.next();
log.info("Removed Key", next.getValue());
}
On Thu, Jul 2, 2015 at 5:36 PM, Shekar Tippur wrote:
> Yi,
>
> There is no exception
Yi,
There is no exception. I want to do couple of things in the window.
- Get all the keys and values and publish to another store (like graphite)
as a list
- Remove all entries.
I can iterate thro the list later but I want to be able to get all kv
values and delete all of them in an atomic oper
Hi, Shekar,
Sorry I was not able to follow up w/ you in time. It is great that you have
found the configure problem and made it work!
As for the exception on the iterator, could you send us the log w/ the
exception?
Thanks!
-Yi
On Thu, Jul 2, 2015 at 4:36 PM, Shekar Tippur wrote:
> Yi,
>
> L
Yi,
Looks like it is working now. There was a redundant line in the config.
I am able to initialize kv store and add values.
In the window code, I am unable to retrieve them and mark them as 0.
Here is my window code:
public void window(MessageCollector collector,
TaskCoordinator coordin
Yi,
Here is my config file:
http://pastebin.com/Kf3C9E0h
- S
Hi, Shekar,
Could you attach the complete config file here? It would be hard just to
debug through snippets from your configure file.
Thanks!
-Yi
On Wed, Jul 1, 2015 at 5:59 PM, Shekar Tippur wrote:
> Sorry, after re-reading the docs,
>
> https://samza.apache.org/learn/documentation/0.8/jobs/
Sorry, after re-reading the docs,
https://samza.apache.org/learn/documentation/0.8/jobs/configuration-table.html#stores-key-serde
I have changed it to
# Define serde
stores.store.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
stores.store.key.serde=byte
stores.store.ms
I do have this in init as well...
public void init(Config config, TaskContext context) {
store = (KeyValueStore) context.getStore("store");
}
You are right. These are primitive types but I was trying to address this
exception:
Exception in thread "main" org.apache.samza.SamzaException: Mus
Hi, Shekar,
This is probably what's missing:
private KeyValueStore store;
public void init(Config config, TaskContext context) {
...
store = (KeyValueStore) context.getStore("store");
...
}
And, it looks like your key and value data types for the store are
primitive types, why do you
Sorry .. That was a copy paste issue
This is what I have
stores.store.key.serde=org.apache.samza.serializers.JsonSerdeFactory
stores.store.msg.serde=org.apache.samza.serializers.JsonSerdeFactory
stores.store.changelog=argos.windowchangelog
and this is how i am initializing it
private KeyValu
So do you use the "store-name" as the kv storage name in your StreamTask
code?
Fang, Yan
yanfang...@gmail.com
On Wed, Jul 1, 2015 at 3:41 PM, Shekar Tippur wrote:
> Yan,
>
> yes. I do have it.
>
> - Shekar
>
> On Wed, Jul 1, 2015 at 3:09 PM, Yan Fang wrote:
>
> > Do you have
> >
> > serializer
Yan,
yes. I do have it.
- Shekar
On Wed, Jul 1, 2015 at 3:09 PM, Yan Fang wrote:
> Do you have
>
> serializers.registry.json.class
> =org.apache.samza.serializers.JsonSerdeFactory
>
> in your config file?
>
>
> Fang, Yan
> yanfang...@gmail.com
>
> On Wed, Jul 1, 2015 at 2:59 PM, Shekar Tippur
Do you have
serializers.registry.json.class
=org.apache.samza.serializers.JsonSerdeFactory
in your config file?
Fang, Yan
yanfang...@gmail.com
On Wed, Jul 1, 2015 at 2:59 PM, Shekar Tippur wrote:
> Yi/Milinda,
>
> I am trying to initialize a kv store. I have the following properties
> define
Yi/Milinda,
I am trying to initialize a kv store. I have the following properties
defined:
stores.store-name.key.serde=json
stores.store-name.msg.serde=json
stores.store-name.changelog=argos.windowchangelog
How do I define a key serde as I am getting this exception:
Exception in thread "main"
Yi,
My use case is more of the latter. Your explanation makes sense now. I was
also looking into Milinda's wiki. She has a section for Kafka
partition SimplePartitioner, which is simple enough as well.
Thanks for all the inputs. Let me see what I come up with while
implementing it.
- Shekar
On
Hi, Shekar,
First, I would like to clarify what you meant by sliding window: is it
defined as windows with size N and advance step size of 1 (which means that
windows overlap and each input message would contribute to multiple counts
in different windows)? Or windows with size N and advance step s
Hi Shekar,
Please have a look at [1].
Milinda
[1] https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
On Mon, Jun 29, 2015 at 1:06 PM, Shekar Tippur wrote:
> Benjamin,
>
> Thanks for the explanation. We dont have any specific partition scheme as
> yet. We just have 2 topi
Benjamin,
Thanks for the explanation. We dont have any specific partition scheme as
yet. We just have 2 topics - raw and processed and we use default
partitioning scheme.
Can you share any code snippet so I can understand it better?
- Shekar
Shekar,
You won't be creating a partition per application. By using the application
name as the partitioning key you ensure all events for a given application
are consistently mapped to the same partition. Multiple applications will
be mapped to each partition without any need for a priori knowled
Milinda,
This is a stream of events where I dont know how many applications are
sending events. I need to dynamically create Kafka partitions.
Can you please confirm the flow:
1. New event comes in
2. Check to see if a partition exists for the application. If not create
one.
3. Implement public st
Hi Shekar,
You can use Kafka's partitioning capabilities to partition your stream
based on application. That will make sure events related to a application
will always ended up in same partition. With this you will have multiple
applications in same partition and each partition will be mapped to a
Milinda,
I see that the document you mentioned addresses windowing but I also need
to group by different applications.
ApplicationCount
---
A100
B40
C69
- Shekar
On Fri, Jun 26, 2015 at 11:39 AM, Shekar
Never mind. I see it here:
http://samza.apache.org/learn/documentation/0.8/container/windowing.html
Thanks again Milinda.
- Shekar
On Fri, Jun 26, 2015 at 11:39 AM, Shekar Tippur wrote:
> Thanks Milinda.
> Is this feature available on 0.8 version of Samza?
>
> - Shekar
>
> On Fri, Jun 26, 201
Thanks Milinda.
Is this feature available on 0.8 version of Samza?
- Shekar
On Fri, Jun 26, 2015 at 11:24 AM, Milinda Pathirage
wrote:
> Hi Shekar,
>
> You can use Samza's local storage (
>
> http://samza.apache.org/learn/documentation/0.9/container/state-management.html
> )
> to keep the windo
Hi Shekar,
You can use Samza's local storage (
http://samza.apache.org/learn/documentation/0.9/container/state-management.html)
to keep the window state and windowing (
http://samza.apache.org/learn/documentation/0.9/container/windowing.html)
capabilities to handle the window advancement. During a
Yan,
*What do you mean by "a local cache"? Is it a db like MySQL, something
likeRocksDB, or even just in-memory?*
Local cache as in Redis
*When you say "another topic", is this the topic consumed by the same
Samzajob as your 5-minutes-job, or in a separate job? What is the
relationbetween the
Hi Shekar,
Need a little more clarification.
What do you mean by "a local cache"? Is it a db like MySQL, something like
RocksDB, or even just in-memory?
When you say "another topic", is this the topic consumed by the same Samza
job as your 5-minutes-job, or in a separate job? What is the relatio
Hello,
My apologies if I have raised it earlier.
Here is the use case:
I have a stream that is partitioned based on application name. I want to be
able to count hte number of events happening for that particular
application in the past 5 minutes (sliding window) and update either
another topic or a
53 matches
Mail list logo
