Re: example applications in malhar

Thanks for the suggestions and I am working on the process to migrate the examples with the guidelines you mentioned. I will send out a list of examples and the destination modules very soon. On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise wrote: > Maybe a good first step

[jira] [Commented] (APEXCORE-526) Publish javadoc for releases on ASF infrastructure

[ https://issues.apache.org/jira/browse/APEXCORE-526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631520#comment-15631520 ] Munagala V. Ramanath commented on APEXCORE-526: ---

[jira] [Comment Edited] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630848#comment-15630848 ] David Yan edited comment on APEXCORE-570 at 11/2/16 11:11 PM: --

[jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630848#comment-15630848 ] David Yan commented on APEXCORE-570: [~PramodSSImmaneni] Maybe we can't block it but can't we

[jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630806#comment-15630806 ] Pramod Immaneni commented on APEXCORE-570: -- [~davidyan] that would lead to deadlock because old

[jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630749#comment-15630749 ] Munagala V. Ramanath commented on APEXCORE-570: --- I think part of the problem is that

[jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630694#comment-15630694 ] David Yan commented on APEXCORE-570: If the spooling capacity limit is reached, would the operator

Re: [DISCUSSION] Custom Control Tuples

Suppose I am processing data in a file and I want to do something at the end of a file at the output operator, I would send an end file control tuple and act on it when I receive it at the output. In a single window I may end up processing multiple files and if I don't have multiple ports and

Re: [DISCUSSION] Custom Control Tuples

With option 2, users can still do idempotent processing by delaying their processing of the control tuples to end window. They have the flexibility with this option. In the usual scenarios, you will have one port and given that control tuples will be sent to all partitions, all the data sent

Re: [DISCUSSION] Custom Control Tuples

The use cases listed in the original discussion don't call for option 2. It seems to come with additional complexity and implementation cost. Can those in favor of option 2 please also provide the use case for it. Thanks, Thomas On Wed, Nov 2, 2016 at 10:36 PM, Siyuan Hua

Re: [DISCUSSION] Custom Control Tuples

I will vote for approach 1. First of all that one sounds easier to do to me. And I think idempotency is important. It may run at the cost of higher latency but I think it is ok And in addition, when in the future if users do need realtime control tuple processing, we can always add the option on

Re: [DISCUSSION] Custom Control Tuples

Pramod, To answer your questions, the control tuples will be delivered to all downstream partitions, and an additional emitControl method (actual name TBD) can be added to DefaultOutputPort without breaking backward compatibility. Also, to clarify, each operator should have the ability to block

[jira] [Commented] (APEXMALHAR-2321) Improve Buckets memory management

[ https://issues.apache.org/jira/browse/APEXMALHAR-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630594#comment-15630594 ] ASF GitHub Bot commented on APEXMALHAR-2321: GitHub user brightchen opened a pull

[GitHub] apex-malhar pull request #481: APEXMALHAR-2321 #resolve #comment Improve Buc...

GitHub user brightchen opened a pull request: https://github.com/apache/apex-malhar/pull/481 APEXMALHAR-2321 #resolve #comment Improve Buckets memory management You can merge this pull request into a Git repository by running: $ git pull

Re: [DISCUSSION] Custom Control Tuples

As a rule of thumb in any real time operating system, control tuples should always be handled using Priority Queues. We may try to control priorities by defining levels. And shall not be delivered at window boundaries. In short, control tuples shall never be treated as any other tuples in real

Re: [DISCUSSION] Custom Control Tuples

I like approach 2 as it gives more flexibility and also allows for low-latency options. I think the following are important as well. 1. Delivering control tuples to all downstream partitions. 2. What mechanism will the operator developer use to send the control tuple? Will it be an additional

Re: [DISCUSSION] Custom Control Tuples

Hi all, I would like to renew the discussion of control tuples. Last time, we were in a debate about whether: 1) the platform should enforce that control tuples are delivered at window boundaries only or: 2) the platform should deliver control tuples just as other tuples and it's the operator

[jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630269#comment-15630269 ] Pramod Immaneni commented on APEXCORE-570: -- For the inter-process case, if the downstream

[jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630072#comment-15630072 ] Thomas Weise commented on APEXCORE-570: --- >From the JIRA description, this sounds like a

[jira] [Commented] (APEXMALHAR-2274) AbstractFileInputOperator gets killed when there are a large number of files.

[ https://issues.apache.org/jira/browse/APEXMALHAR-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629893#comment-15629893 ] Matt Zhang commented on APEXMALHAR-2274: The scanner in FileSplitterInput is more complicate.

[jira] [Created] (APEXMALHAR-2326) Failures in SQS unit tests

Sanjay M Pujare created APEXMALHAR-2326: --- Summary: Failures in SQS unit tests Key: APEXMALHAR-2326 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2326 Project: Apache Apex Malhar

[GitHub] apex-malhar pull request #480: [APEXMALHAR-2220] Move the FunctionOperator t...

GitHub user d9liang opened a pull request: https://github.com/apache/apex-malhar/pull/480 [APEXMALHAR-2220] Move the FunctionOperator to Malhar library Merge function operators under org.apache.apex.malhar.stream.api.operator and function interface under

[jira] [Commented] (APEXMALHAR-2220) Move the FunctionOperator to Malhar library

[ https://issues.apache.org/jira/browse/APEXMALHAR-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629562#comment-15629562 ] ASF GitHub Bot commented on APEXMALHAR-2220: GitHub user d9liang opened a pull request:

[jira] [Resolved] (APEXMALHAR-2302) Exposing few properties of FSSplitter and BlockReader operators to FSRecordReaderModule to tune Application

[ https://issues.apache.org/jira/browse/APEXMALHAR-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yogi Devendra resolved APEXMALHAR-2302. --- Resolution: Fixed Fix Version/s: 3.6.0 > Exposing few properties of

[GitHub] apex-malhar pull request #457: APEXMALHAR-2302 Exposing few properties of FS...

Github user asfgit closed the pull request at: https://github.com/apache/apex-malhar/pull/457 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Commented] (APEXMALHAR-2302) Exposing few properties of FSSplitter and BlockReader operators to FSRecordReaderModule to tune Application

[ https://issues.apache.org/jira/browse/APEXMALHAR-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629394#comment-15629394 ] ASF GitHub Bot commented on APEXMALHAR-2302: Github user asfgit closed the pull request

[jira] [Commented] (APEXMALHAR-2325) Same block id is emitting from FSInputModule

[ https://issues.apache.org/jira/browse/APEXMALHAR-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628804#comment-15628804 ] ASF GitHub Bot commented on APEXMALHAR-2325: GitHub user chaithu14 opened a pull request:

[GitHub] apex-malhar pull request #479: APEXMALHAR-2325 1) Set the file system defaul...

GitHub user chaithu14 opened a pull request: https://github.com/apache/apex-malhar/pull/479 APEXMALHAR-2325 1) Set the file system default block size to the reader. 2) Set the block size to the reader context You can merge this pull request into a Git repository by running:

[jira] [Created] (APEXMALHAR-2325) Same block id is emitting from FSInputModule

Chaitanya created APEXMALHAR-2325: - Summary: Same block id is emitting from FSInputModule Key: APEXMALHAR-2325 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2325 Project: Apache Apex Malhar

[jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628460#comment-15628460 ] Pramod Immaneni commented on APEXCORE-570: -- Here is an example on how to do this from within

[jira] [Created] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow

Pramod Immaneni created APEXCORE-570: Summary: Prevent upstream operators from getting too far ahead when downstream operators are slow Key: APEXCORE-570 URL:

Re: [jira] [Commented] (APEXMALHAR-2303) S3 Line By Line Module

Hi Apex Dev Community, For Fixed Width S3 record Reader, the input is the block metadata containing the block offset and the block length. The length of the block may not be a factor of the length of the record. (For eg, block length can be 1MB, record length can be 23 bytes) Hence, the first