date:20160829

Re: [VOTE] Apache Apex Malhar Release 3.5.0 (RC1)

2016-08-29 Thread Thomas Weise

Hi Justin,

Thanks for the thorough inspection. Please see comments below.

David, Siyuan, there are questions for you as well.

Thanks,
Thomas


On Mon, Aug 29, 2016 at 7:16 PM, Justin Mclean 
wrote:

> Hi,
>
> Sorry but -1 until license/copyright issues resolved.
>
> I checked:
> - signature and hashes good
> - LICENSE and NOTICE good
> - release contains content not compatible with Apache license (see below)
> - no unexpected binary files
> - all source files have ASF headers
>
> I’m fairly sure that bundling this [1] is not allowed. Project Gutenberg
> is generally considered public domain but it not listed as an Apache
> compatible license here [3] and it is not an OSI approved license [4]. May
> be best to ask on legal discuss for clarification.
>
> You may also want to replace the content of [2] with something else as it
> copyright Time magazine. (I thought this has been raised before?). These
> files [5][6][7] also look to have similar copyright issues.
>
> Thanks,
> Justin
>
> 1. apache-apex-malhar-3.5.0/library/src/test/resources/wordcount.txt
>

Author: David Yan 
Date:   Thu Jun 23 14:37:29 2016 -0700

David, did you find something that indicates that we can use this?


> 2. apache-apex-malhar-3.5.0/demos/wordcount/src/main/
> resources/samplefile.txt
>

There was a file with same content that was removed in 3.2.0 and you
brought it up back then. It turns out that this existed in another
location.. ouch. I will submit a change to remove the content.


> 3. http://www.apache.org/legal/resolved.html#category-a
> 4. https://opensource.org/licenses/alphabetical
> 5. apache-apex-malhar-3.5.0/demos/highlevelapi/src/test/
> resources/sampletweets.txt
> 6 apache-apex-malhar-3.5.0/stream/src/test/resources/sampletweets.txt
>

commit 266b04116760dbd4d5cad6b4102b06153ac96a5f
Author: Siyuan Hua 
Date:   Tue Jul 12 11:57:09 2016 -0700


> 7. apache-apex-malhar-3.5.0/contrib/src/test/resources/
> com/datatorrent/contrib/romesyndication/datatorrent_feed_updated.rss
> 8. apache-apex-malhar-3.5.0/contrib/src/test/resources/
> com/datatorrent/contrib/romesyndication/datatorrent_feed.rss


These files (7./8.) were already replaced when the issue first came up with
3.2.0, here is the diff:

https://github.com/apache/apex-malhar/commit/75dc74fe33740cd6d2e3432a4248d95333412648

Content is from DT blog site, do you see an issue with it?

[jira] [Comment Edited] (APEXMALHAR-2130) implement scalable windowed storage

2016-08-29 Thread Chandni Singh (JIRA)

[
https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15447840#comment-15447840
]

Chandni Singh edited comment on APEXMALHAR-2130 at 8/30/16 3:04 AM:

Note: The main change in ManagedState which is required here is that
timeBuckets (Window time in your example) is now computed outside ManagedState.
TimeBuckets were being computed by TimeBucketAssigner within ManagedState but
now it will be provided to it.

Since event time is arbitrary, unlike processing time, the actual key
representing the timebucket cannot be assumed a natural sequence. However,
TimeBucketAssigner.getTimeBucketAndAdjustBoundaries seems to return a long that
is sequential starting from 0. We want to make the actual timebucket key based
on the actual event window timestamp. Chandni Singh Will this break anything?

Answer: No it will not break anything. The time here is event time and this
does NOT assume that events are received in order. Based on event time, this
method creates timebucket. In your use case, the time bucket is computed
outside ManagedState so there are 2 ways to approach it:
- create a special TimeBucketAssigner which will just return the input Window
for the event. It will not further compute timebucket.
- make TimeBucketAssigner an optional property in AbstractManagedStateImpl. If
it is null, then the time argument is used as timebucket save in Bucket.

>>>
Expiring and purging are done very differently and should be based on 3.
Managed State should determine whether to purge a timebucket based on whether
an Apex window is committed and whether all event windows that belong to that
timebucket are marked "deleted" for that Apex window.

Answer: This is handled by TimeBucketAssigner again. I don't think much change
is needed here. TimeBucketAssigner computes a timeBucket (in your case, this
corresponds to Window time) and checks if the oldest buckets need to be purged
(line 132 - 133). It figures out the lowest purgeable timebucket. In the
endWindow, it informs IncrementalCheckpointManager, that it can delete all the
timebuckets<=lowestPurgeableTimeBucket. However, IncrementalCheckpointManager
deletes the data up to that timebucket only when the window in which it was
request to be purged gets committed. So this will remain the same for you as
well.

I think this can also by achieved by creating a special TimeBucketAssigner and
overriding a few methods.

was (Author: csingh):
Note: The main change in ManagedState which is required here is that
timeBuckets (Window time in your example) is now computed outside ManagedState.
TimeBuckets were being computed by TimeBucketAssigner within ManagedState but
now it will be provided to it.

Expiring and purging are done very differently and should be based on 3.
Managed State should determine whether to purge a timebucket based on whether
an Apex window is committed and whether all event windows that belong to that
timebucket are marked "deleted" for that Apex window.

39 matches

Mail list logo