Re: Dynamic partition is not working in Kafka Input Operator

Hi Sandesh, No. Kafka Input Operator(0.8 version) supports dynamic partition based on Kafka partitions. Created a JIRA (APEXCORE-494 ) for tracking this issue. Regards, Chaitanya On Tue, Jul 19, 2016 at 9:47 AM, Sandesh Hegde wrote

[jira] [Updated] (APEXCORE-494) Window id of downstream operator is not moving after dynamic partition of upstream operator.

[ https://issues.apache.org/jira/browse/APEXCORE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaitanya updated APEXCORE-494: --- Attachment: PhysicalDAGBeforeKafkaRepartition.png PhysicalDAGAfterKafkaRepartition.pn

[jira] [Created] (APEXCORE-494) Window id of downstream operator is not moving after dynamic partition of upstream operator.

Chaitanya created APEXCORE-494: -- Summary: Window id of downstream operator is not moving after dynamic partition of upstream operator. Key: APEXCORE-494 URL: https://issues.apache.org/jira/browse/APEXCORE-494

[jira] [Created] (APEXMALHAR-2160) Add Enum Support in CSV Formatter

shubham pathak created APEXMALHAR-2160: -- Summary: Add Enum Support in CSV Formatter Key: APEXMALHAR-2160 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2160 Project: Apache Apex Malhar

[jira] [Created] (APEXMALHAR-2159) Add Enum Support in CSV Parser

shubham pathak created APEXMALHAR-2159: -- Summary: Add Enum Support in CSV Parser Key: APEXMALHAR-2159 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2159 Project: Apache Apex Malhar

Re: Container & memory resource allocation

Hi Pradeep, This is a great content to add to the documents. These are the common set of errors which might get googled and hence great to get indexed as well. You can take a look at: https://github.com/apache/apex-core/tree/master/docs The docs for apex reside there in markdown format. Probably

[GitHub] apex-core pull request #357: APEXCORE-488: Issues in SSL communication with ...

Github user asfgit closed the pull request at: https://github.com/apache/apex-core/pull/357 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[jira] [Commented] (APEXCORE-488) Issues in SSL communication with StrAM

[ https://issues.apache.org/jira/browse/APEXCORE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383621#comment-15383621 ] ASF GitHub Bot commented on APEXCORE-488: - Github user asfgit closed the pull re

Operator name in BatchedOperatorStats interface.

Hi All, We support shared stats listener, But user do not have any way to identify the operator for which the stat listener is called. The BatchedOperatorStats only provides operatorId, and there is no api available to use to get operator information from operatorId. Can we include operator name

Container & memory resource allocation

Container & memory resource allocation has been a common question around and so I thought it would be good to explain related configuration parameters. Please feel free to let me know your thoughts. Also I'm planning to add following set of information under Apex Docs. How could one add this to A

Re: condition in FilterOperator

Hi Yogi, This is something that needs to be fixed in Filter operator. I see at line 130 in filter operator that the expected return type is Boolean.class.. I think it should be boolean.class, unless there was some other reason why it was made like that. I would prefer not to add implicit pair of

[jira] [Commented] (APEXMALHAR-1701) Deduper : create a deduper backed by Managed State

[ https://issues.apache.org/jira/browse/APEXMALHAR-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383562#comment-15383562 ] ASF GitHub Bot commented on APEXMALHAR-1701: Github user bhupeshchawda co

[GitHub] apex-malhar pull request #335: [Review Only] APEXMALHAR-1701 Deduper with Ma...

Github user bhupeshchawda commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/335#discussion_r71270785 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/dedup/AbstractDeduper.java --- @@ -0,0 +1,463 @@ +/** + * Licensed to the Apache

Re: Dynamic partition is not working in Kafka Input Operator

Was this resolved? My understanding is that, Kafka Input operator doesn't support the changes in Kafka partitions after the initial launch. On Mon, Jul 18, 2016 at 1:54 AM Chaitanya Chebolu wrote: > Hi All, > >I am facing dynamic partition issues in 0.8 version of Kafka Input > Operator. My

[jira] [Commented] (APEXMALHAR-1701) Deduper : create a deduper backed by Managed State

[ https://issues.apache.org/jira/browse/APEXMALHAR-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383557#comment-15383557 ] ASF GitHub Bot commented on APEXMALHAR-1701: Github user bhupeshchawda co

[jira] [Commented] (APEXMALHAR-1701) Deduper : create a deduper backed by Managed State

[ https://issues.apache.org/jira/browse/APEXMALHAR-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383558#comment-15383558 ] ASF GitHub Bot commented on APEXMALHAR-1701: Github user bhupeshchawda co

[GitHub] apex-malhar pull request #335: [Review Only] APEXMALHAR-1701 Deduper with Ma...

Github user bhupeshchawda commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/335#discussion_r71270664 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/dedup/AbstractDeduper.java --- @@ -0,0 +1,463 @@ +/** + * Licensed to the Apache

[GitHub] apex-malhar pull request #335: [Review Only] APEXMALHAR-1701 Deduper with Ma...

Github user bhupeshchawda commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/335#discussion_r71270656 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/dedup/AbstractDeduper.java --- @@ -0,0 +1,463 @@ +/** + * Licensed to the Apache

Re: auto-generated emails

Pramod, My Suggestion is on the same lines as what Yogi suggested, except instead of creating a seperate mailing list, send further mails to folks in the comments. -Chinmay. On Tue, Jul 19, 2016 at 3:07 AM, Pramod Immaneni wrote: > Chinmay are you suggesting the first mail only sent to folks

Re: auto-generated emails

Chinmay are you suggesting the first mail only sent to folks mentioned and not everyone in the list? I like Yogi's suggestion on having a separate commits list where all commit emails go but initial emails are still sent to dev and dev can mostly focus on discussions. Anyone, wants to volunteer to

[GitHub] apex-site pull request #42: Added updating documentation to release process

Github user PramodSSImmaneni commented on a diff in the pull request: https://github.com/apache/apex-site/pull/42#discussion_r71226364 --- Diff: src/md/release.md --- @@ -150,6 +150,47 @@ mkdir svn-dist && cp *-source-* svn-dist/ svn import svn-dist https://dist.apache.org/rep

[GitHub] apex-site pull request #42: Added updating documentation to release process

Github user PramodSSImmaneni commented on a diff in the pull request: https://github.com/apache/apex-site/pull/42#discussion_r71226057 --- Diff: src/md/release.md --- @@ -150,6 +150,47 @@ mkdir svn-dist && cp *-source-* svn-dist/ svn import svn-dist https://dist.apache.org/rep

[GitHub] apex-malhar pull request #345: REVIEW ONLY (WindowedOperator): splitting Win...

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/345#discussion_r71225979 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/window/impl/SpillableWindowedKeyedStorage.java --- @@ -0,0 +1,136 @@ +package org.apac

[GitHub] apex-malhar pull request #345: REVIEW ONLY (WindowedOperator): splitting Win...

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/345#discussion_r71225716 --- Diff: library/src/main/java/org/apache/apex/malhar/lib/window/impl/SpillableWindowedKeyedStorage.java --- @@ -0,0 +1,136 @@ +package org.apac

[jira] [Commented] (APEXMALHAR-2158) Duplication of data emitted when the Kafka Input Operator(0.8 version) redeploys

[ https://issues.apache.org/jira/browse/APEXMALHAR-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382108#comment-15382108 ] ASF GitHub Bot commented on APEXMALHAR-2158: GitHub user chaithu14 opened

[GitHub] apex-malhar pull request #346: APEXMALHAR-2158 Fixed the duplication of mess...

GitHub user chaithu14 opened a pull request: https://github.com/apache/apex-malhar/pull/346 APEXMALHAR-2158 Fixed the duplication of messages emitted issue when the Kafka Input operator redeployed You can merge this pull request into a Git repository by running: $ git pull ht

[jira] [Created] (APEXMALHAR-2158) Duplication of data emitted when the Kafka Input Operator(0.8 version) redeploys

Chaitanya created APEXMALHAR-2158: - Summary: Duplication of data emitted when the Kafka Input Operator(0.8 version) redeploys Key: APEXMALHAR-2158 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2158

Dynamic partition is not working in Kafka Input Operator

Hi All, I am facing dynamic partition issues in 0.8 version of Kafka Input Operator. My application has the following DAG: KafkaSinglePortStringInputOperator(Input) -> ConsoleOutputOperator(Output) I launched the application with below configuration: Kafka topic created with single part

Improvements in JSON Formatter

Hello Community, I would like to make following improvements in existing JSON Formatter *1. Ability to provide date format for different date fields in POJO*. Current implementation accepts just one format and applies the same to all date fields. *2*. *Ability to provide JSON field to POJO field m

Fwd: APEXMALHAR-1701 Deduper in Malhar

Yes, in that case, it can be modelled as a session window. A session for every key can start with the first occurrence of that key and lasts for some specific time duration. This leads to another question: How exactly do we want to model Dedup expiry? - Are the windows fixed on the event time

Re: APEXMALHAR-1701 Deduper in Malhar

+1 on the suggested way forward No clear why you say the windows are fixed though. What if I want the dedup to happen based on the most recent event with a given key + n time units? On Mon, Jul 18, 2016 at 9:05 AM, Bhupesh Chawda wrote: > I can see that Dedup seems like a case where state is c

Re: APEXMALHAR-1701 Deduper in Malhar

Dear Community, +1 to Bhupesh's suggestion. I would suggest to go ahead with the Managed State and once we have proper analysis on the windowed operator + large storage backed windowed operator - we should implement operators such as dedup with it. Regards, Mohit On Mon, Jul 18, 2016 at 12:35 P

[jira] [Created] (APEXMALHAR-2157) Improvements in JSON Formatter

shubham pathak created APEXMALHAR-2157: -- Summary: Improvements in JSON Formatter Key: APEXMALHAR-2157 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2157 Project: Apache Apex Malhar

Fwd: APEXMALHAR-1701 Deduper in Malhar

I can see that Dedup seems like a case where state is continuously merged with older state. State in this case is the set of unique tuples. However, for Dedup use case, the event windows are, in a way, fixed, and do not depend on the incoming tuples. In-coming tuples are just *assigned* to these wi