Thank you very much for your reply , Daniel
On Thu, Dec 8, 2016 at 7:07 PM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote:
> There's no need to extend Spark's API, look at mapWithState for examples.
>
> On Thu, Dec 8, 2016 at 4:49 AM, Anty Rao wrote:
>
>>
>>
>> On Wed, Dec 7, 2016 at 7:4
There's no need to extend Spark's API, look at mapWithState for examples.
On Thu, Dec 8, 2016 at 4:49 AM, Anty Rao wrote:
>
>
> On Wed, Dec 7, 2016 at 7:42 PM, Anty Rao wrote:
>
>> Hi
>> I'm new to Spark. I'm doing some research to see if spark streaming can
>> solve my problem. I don't want to
On Wed, Dec 7, 2016 at 7:42 PM, Anty Rao wrote:
> Hi
> I'm new to Spark. I'm doing some research to see if spark streaming can
> solve my problem. I don't want to keep per-key state,b/c my data set is
> very huge and keep a little longer time, it not viable to keep all per key
> state in memory.I
@Daniel
Thanks for your reply. I will try it.
On Wed, Dec 7, 2016 at 8:47 PM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote:
> Hi Anty,
> What you could do is keep in the state only the existence of a key and
> when necessary pull it from a secondary state store like HDFS or HBASE.
>
> Da
Hi Anty,
What you could do is keep in the state only the existence of a key and when
necessary pull it from a secondary state store like HDFS or HBASE.
Daniel
On Wed, Dec 7, 2016 at 1:42 PM, Anty Rao wrote:
> Hi
> I'm new to Spark. I'm doing some research to see if spark streaming can
> solve m
Hi
I'm new to Spark. I'm doing some research to see if spark streaming can
solve my problem. I don't want to keep per-key state,b/c my data set is
very huge and keep a little longer time, it not viable to keep all per key
state in memory.Instead, i want to have a bloom filter based state. Does it
p