[jira] [Commented] (APEXMALHAR-2126) Suggest: Share Slice Buffer

2016-06-29 Thread Vlad Rozov (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356525#comment-15356525 ] Vlad Rozov commented on APEXMALHAR-2126: Why is it necessary to have Slice he

[jira] [Commented] (APEXMALHAR-2069) FileSplitterInput and TimeBasedDirectoryScanner - move operational fields initialization from constructor to setup

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356375#comment-15356375 ] ASF GitHub Bot commented on APEXMALHAR-2069: Github user vrozov commented

[GitHub] apex-malhar pull request #329: Fix for APEXMALHAR-2069: moved creation of sc...

2016-06-29 Thread vrozov
Github user vrozov commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/329#discussion_r69063970 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java --- @@ -318,6 +317,9 @@ public TimeBasedDirectoryScanner() @Over

[jira] [Commented] (APEXMALHAR-2130) implement scalable windowed storage

2016-06-29 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356342#comment-15356342 ] Thomas Weise commented on APEXMALHAR-2130: -- What's the use case? I think thi

[jira] [Commented] (APEXMALHAR-2069) FileSplitterInput and TimeBasedDirectoryScanner - move operational fields initialization from constructor to setup

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356287#comment-15356287 ] ASF GitHub Bot commented on APEXMALHAR-2069: GitHub user sanjaypujare ope

[GitHub] apex-malhar pull request #329: Fix for APEXMALHAR-2069: moved creation of sc...

2016-06-29 Thread sanjaypujare
GitHub user sanjaypujare opened a pull request: https://github.com/apache/apex-malhar/pull/329 Fix for APEXMALHAR-2069: moved creation of scanService to setup, @vro… @vrozov pls review. You can merge this pull request into a Git repository by running: $ git pull https://gith

Download counts

2016-06-29 Thread Sasha Parfenov
Is there a way to get results for number of downloads by version of Apache Apex from apex.apache.org? Doesn't matter if it's done via Google Analytics, or by looking at web server logs. Just wanted to know if this is possible, how to get access/display this data on the site, and how other Apache

[jira] [Commented] (APEXMALHAR-2130) implement scalable windowed storage

2016-06-29 Thread bright chen (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356232#comment-15356232 ] bright chen commented on APEXMALHAR-2130: - Here are some of thoughts: There a

[jira] [Created] (APEXMALHAR-2130) implement scalable windowed storage

2016-06-29 Thread bright chen (JIRA)
bright chen created APEXMALHAR-2130: --- Summary: implement scalable windowed storage Key: APEXMALHAR-2130 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130 Project: Apache Apex Malhar

[jira] [Commented] (APEXMALHAR-2116) File Record reader module

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355935#comment-15355935 ] ASF GitHub Bot commented on APEXMALHAR-2116: Github user amberarrow comme

[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...

2016-06-29 Thread amberarrow
Github user amberarrow commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/326#discussion_r69042882 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java --- @@ -0,0 +1,332 @@ +/** + * Licensed to the Apache

[jira] [Commented] (APEXMALHAR-2116) File Record reader module

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355930#comment-15355930 ] ASF GitHub Bot commented on APEXMALHAR-2116: Github user amberarrow comme

[jira] [Commented] (APEXMALHAR-2116) File Record reader module

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355929#comment-15355929 ] ASF GitHub Bot commented on APEXMALHAR-2116: Github user amberarrow comme

[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...

2016-06-29 Thread amberarrow
Github user amberarrow commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/326#discussion_r69042391 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java --- @@ -0,0 +1,332 @@ +/** + * Licensed to the Apache

[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...

2016-06-29 Thread amberarrow
Github user amberarrow commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/326#discussion_r69042316 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java --- @@ -0,0 +1,332 @@ +/** + * Licensed to the Apache

[jira] [Commented] (APEXMALHAR-2116) File Record reader module

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355924#comment-15355924 ] ASF GitHub Bot commented on APEXMALHAR-2116: Github user amberarrow comme

[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...

2016-06-29 Thread amberarrow
Github user amberarrow commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/326#discussion_r69041055 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java --- @@ -0,0 +1,332 @@ +/** + * Licensed to the Apache

Re: Why the setSink method takes Object instead of Type

2016-06-29 Thread Pramod Immaneni
I believe it's because different types of sinks can get plugged in depending on how the operators are deployed. If the operator and its downstream operator are in same container it is a queue otherwise it is a publisher to buffer server that is serializing the data generically independent of the da

Re: Why the setSink method takes Object instead of Type

2016-06-29 Thread Amol Kekre
Siyuan, I am not able to remember right now. Chetan or Thomas may remember. Even casting a tuple from upstream port or to downstream port it should work if both the above apis are . As of today, I cannot see a reason why we did this. Thks Amol On Wed, Jun 29, 2016 at 1:41 PM, Siyuan Hua wrote:

Why the setSink method takes Object instead of Type

2016-06-29 Thread Siyuan Hua
The setSink method on OutputPort takes Sink But the getSink method on InputPort returns a Sink Is there a reason for this asymmetric API ? Regards, Siyuan

[jira] [Commented] (APEXMALHAR-2129) ManagedState: Add a disable purging option

2016-06-29 Thread Chandni Singh (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355507#comment-15355507 ] Chandni Singh commented on APEXMALHAR-2129: --- [~bhupesh] I re-opened the tic

[jira] [Updated] (APEXMALHAR-2129) ManagedState: Add a disable purging option

2016-06-29 Thread Chandni Singh (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated APEXMALHAR-2129: -- Description: Have an option that can disable purging of data. (was: TimeBucketAssi

[jira] [Updated] (APEXMALHAR-2129) ManagedState: Add a disable purging option

2016-06-29 Thread Chandni Singh (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated APEXMALHAR-2129: -- Priority: Major (was: Minor) > ManagedState: Add a disable purging option > --

[jira] [Updated] (APEXMALHAR-2129) ManagedState: Add a disable purging option

2016-06-29 Thread Chandni Singh (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated APEXMALHAR-2129: -- Summary: ManagedState: Add a disable purging option (was: Introduce option to adva

[jira] [Reopened] (APEXMALHAR-2129) ManagedState: Add a disable purging option

2016-06-29 Thread Chandni Singh (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh reopened APEXMALHAR-2129: --- Assignee: Chandni Singh (was: Bhupesh Chawda) > ManagedState: Add a disable pu

Re: APEXMALHAR-1701 Deduper in Malhar

2016-06-29 Thread Bhupesh Chawda
Hi Thomas, I agree that the case of processing bounded data is a special case of unbounded data. Th difference I was pointing out was in terms of expiry. This is not applicable in case of bounded data sets, while unbounded data sets will inherently use expiry for limiting the amount of data to be

[GitHub] apex-malhar pull request #294: APEXMALHAR-2076 #resolve #comment add Abstrac...

2016-06-29 Thread siyuanh
Github user siyuanh commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/294#discussion_r68977507 --- Diff: contrib/src/main/java/com/datatorrent/contrib/kafka/AbstractTupleUniqueExactlyOnceKafkaOutputOperator.java --- @@ -0,0 +1,610 @@ +/**

Re: APEXMALHAR-1701 Deduper in Malhar

2016-06-29 Thread Thomas Weise
Bhupesh, Why is there a distinction between bounded and unbounded data? I see the former as a special case of the latter? When rewinding the stream or reprocessing the stream in another run the operator should produce the same result. This operator should be idempotent also. That implies that co

[jira] [Commented] (APEXMALHAR-2129) Introduce option to advance time through Expiry task in TimeBucketAssigner

2016-06-29 Thread Chandni Singh (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355324#comment-15355324 ] Chandni Singh commented on APEXMALHAR-2129: --- I am not disagreeing with your

[jira] [Commented] (APEXMALHAR-2116) File Record reader module

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355202#comment-15355202 ] ASF GitHub Bot commented on APEXMALHAR-2116: GitHub user yogidevendra reo

[jira] [Commented] (APEXMALHAR-2116) File Record reader module

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355201#comment-15355201 ] ASF GitHub Bot commented on APEXMALHAR-2116: Github user yogidevendra clo

[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...

2016-06-29 Thread yogidevendra
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/326 APEXMALHAR-2116 Added FS record reader operator, module, test You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malh

[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...

2016-06-29 Thread yogidevendra
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/326 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feat

[jira] [Commented] (APEXMALHAR-2116) File Record reader module

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355113#comment-15355113 ] ASF GitHub Bot commented on APEXMALHAR-2116: Github user yogidevendra clo

[jira] [Commented] (APEXMALHAR-2116) File Record reader module

2016-06-29 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355114#comment-15355114 ] ASF GitHub Bot commented on APEXMALHAR-2116: GitHub user yogidevendra reo

[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...

2016-06-29 Thread yogidevendra
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/326 APEXMALHAR-2116 Added FS record reader operator, module, test You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malh

[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...

2016-06-29 Thread yogidevendra
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/326 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feat

[jira] [Closed] (APEXMALHAR-2129) Introduce option to advance time through Expiry task in TimeBucketAssigner

2016-06-29 Thread Bhupesh Chawda (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda closed APEXMALHAR-2129. -- Resolution: Invalid > Introduce option to advance time through Expiry task in TimeB

[jira] [Commented] (APEXMALHAR-2129) Introduce option to advance time through Expiry task in TimeBucketAssigner

2016-06-29 Thread Bhupesh Chawda (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355051#comment-15355051 ] Bhupesh Chawda commented on APEXMALHAR-2129: Okay. I agree about the boun

Re: APEXMALHAR-1701 Deduper in Malhar

2016-06-29 Thread Bhupesh Chawda
Hi All, I want to validate the use cases for de-duplication that will be going as part of this implementation. - *Bounded data set* - This is de-duplication for bounded data. For example, data sets which are old or fixed or which may not have a time field at all. Example: Las

[GitHub] apex-malhar pull request #324: Spillable Datastructures PR for review only

2016-06-29 Thread chaithu14
Github user chaithu14 commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/324#discussion_r68926798 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/state/spillable/SpillableByteArrayListMultimapImpl.java --- @@ -0,0 +1,241 @@ +package

[GitHub] apex-malhar pull request #324: Spillable Datastructures PR for review only

2016-06-29 Thread chaithu14
Github user chaithu14 commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/324#discussion_r68926769 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/state/spillable/SpillableByteArrayListMultimapImpl.java --- @@ -0,0 +1,241 @@ +package

[jira] [Commented] (APEXMALHAR-2129) Introduce option to advance time through Expiry task in TimeBucketAssigner

2016-06-29 Thread Chandni Singh (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354726#comment-15354726 ] Chandni Singh commented on APEXMALHAR-2129: --- [~bhupesh] Purging is tied at