[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-06 Thread Chandni Singh (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552532#comment-15552532
 ] 

Chandni Singh commented on APEXMALHAR-2276:
---

I don't see a problem. In Scenario A, it is evident right at the time of 
insertion that update is old and is ignored.
In Scenario B, it is unknown at the time of insertion that update is old.

Which is why I mentioned, that the client needs to do a get() and then put(). 

ManagedStateImpl and ManagedTimeStateImpl are not for maintaining time series 
in memory. 

> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-06 Thread Chaitanya (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552495#comment-15552495
 ] 

Chaitanya commented on APEXMALHAR-2276:
---

[~csingh] My query is on insertion. 
In Scenario (A): mt.put(1008, k1, v3) -> This tuple is not inserted into Bucket.
In Scenario (B): mt.put(1008, k1, v3); -> This tuple is successfully inserted 
into Bucket. 


> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-06 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552366#comment-15552366
 ] 

ASF GitHub Bot commented on APEXMALHAR-2276:


Github user chandnisingh closed the pull request at:

https://github.com/apache/apex-malhar/pull/443


> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-06 Thread Chaitanya (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550997#comment-15550997
 ] 

Chaitanya commented on APEXMALHAR-2276:
---

I too agree  But, I have a query relates to Managed State.

 If the same keys are inserted at different timestamps. While searching from 
Bucket, it's not getting the same value. I would like to know, Is there any 
constraint on this?

Example: 
===
ManagedTimeStateImpl mt = new ManagedTimeStateImpl();
bucketSpanTime = 0.5 secs

Scenario (a):
mt.put(1004, k1, v1);
mt.put(1505, k1, v2);
mt.put(1008, k1, v3);
---
checkpoint()

mt.get(k1).equals(v2) will return true.

Scenario (b):

mt.put(1004, k1, v1);
mt.put(1505, k1, v2);

checkpoint()
mt.put(1008, k1, v3);
mt.get(k1).equals(v2) will return false.


> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-05 Thread Chandni Singh (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550649#comment-15550649
 ] 

Chandni Singh commented on APEXMALHAR-2276:
---

I agree with [~thw] too. This creates un-necessary overhead for every use case. 
 Additional space for timestamp is required per key both in memory and disk. 
While making the changes though, I discovered a race condition and an issue 
with free space. I will create another ticket for that.

> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-05 Thread David Yan (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550470#comment-15550470
 ] 

David Yan commented on APEXMALHAR-2276:
---

Agreed with [~thw] here. The windowed operator does not need this case to be 
handled.

> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-05 Thread Thomas Weise (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550331#comment-15550331
 ] 

Thomas Weise commented on APEXMALHAR-2276:
--

Why does this need to be solved in managed state? If time isn't part of the key 
then the operator using it can perform such check when it is really needed by 
reading the previous value and comparing time along with other checks that may 
be needed. I think we should not further complicate the implementation unless 
really needed.


> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550316#comment-15550316
 ] 

ASF GitHub Bot commented on APEXMALHAR-2276:


GitHub user chandnisingh opened a pull request:

https://github.com/apache/apex-malhar/pull/443

APEXMALHAR-2276 over writing the value of key only when the time is g…

…reater than equal to latest time of the entry

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chandnisingh/apex-malhar APEXMALHAR-2276

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #443


commit ea5012c9cb5b17cf6bbde0f73ecd1fd9bca0de94
Author: Chandni Singh 
Date:   2016-10-05T23:39:49Z

APEXMALHAR-2276 over writing the value of key only when the time is greater 
than equal to latest time of the entry




> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-04 Thread Chandni Singh (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546623#comment-15546623
 ] 

Chandni Singh commented on APEXMALHAR-2276:
---

Yes time gets mapped to a time bucket, however the exact value of time is lost 
and currently we cannot derive time from the timebucket. 

This affects cases where we want to save the most recent value of a key. In 
use-cases, like de-duplication/aggregation this is not the case.

In De-duplication we check whether there is a key present and drop the event if 
its duplicate. 
In Aggregation, we use the existing value in the time-bucket and aggregate it 
with new value.


> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-04 Thread Chandni Singh (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546592#comment-15546592
 ] 

Chandni Singh commented on APEXMALHAR-2276:
---

As [~chaithu] pointed out, we need to compare time and not time buckets. 
The fix for that will not be simple. For each key/value now we have to remember 
the latest time. So far bucket didn't have reference to time which means the 
bucket API will probably change as well.


> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-10-04 Thread Siyuan Hua (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546598#comment-15546598
 ] 

Siyuan Hua commented on APEXMALHAR-2276:


Time will be mapped to a time bucket anyways, correct?

> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
> Fix For: 3.6.0
>
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2276) ManagedState: value of a key does not get over-written in the same time bucket

2016-09-30 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536907#comment-15536907
 ] 

ASF GitHub Bot commented on APEXMALHAR-2276:


GitHub user chandnisingh opened a pull request:

https://github.com/apache/apex-malhar/pull/437

APEXMALHAR-2276 over writing the value of a key in the same time bucket

@davidyan74 @ilooner please review and merge since you are familiar with 
ManagedState code 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chandnisingh/apex-malhar APEXMALHAR-2276

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #437


commit ab7bab27c340ed7a33217389f9b1c3f39eac4beb
Author: Chandni Singh 
Date:   2016-09-30T20:01:09Z

APEXMALHAR-2276 over writing the value of a key in the same time bucket




> ManagedState: value of a key does not get over-written in the same time bucket
> --
>
> Key: APEXMALHAR-2276
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2276
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Chandni Singh
>
> For example:
> ManagedTimeUnifiedStateImpl mtus;
> mtus.put(1, key1, val1)
> mtus.put(1, key1, val2)
> mtus.get(1, key1).equals(val2) will return false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)