Re: Input needed for ApexCli.

It could be useful to have a flag and let user decide the best approach fit for him. We can have the default behaviour as best-effort with support for Validate and fail via flag. Ajay On Sat, 5 Aug 2017 at 8:33 AM, Bhupesh Chawda wrote: > +1 for best effort with

[jira] [Commented] (APEXMALHAR-2508) CacheManager throws exception when the operator receives a shutdown request

[ https://issues.apache.org/jira/browse/APEXMALHAR-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048742#comment-16048742 ] Ajay Gupta commented on APEXMALHAR-2508: Exception Stack trace

[jira] [Assigned] (APEXMALHAR-2452) Batch support for JDBC I/O operators

[ https://issues.apache.org/jira/browse/APEXMALHAR-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Gupta reassigned APEXMALHAR-2452: -- Assignee: Ajay Gupta > Batch support for JDBC I/O operat

[jira] [Assigned] (APEXMALHAR-2451) Batch support for File I/O operators

[ https://issues.apache.org/jira/browse/APEXMALHAR-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Gupta reassigned APEXMALHAR-2451: -- Assignee: Ajay Gupta (was: Bhupesh Chawda) > Batch support for File

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

urpose the time component to segregate batches. > > > > 2 possible options that come to mind: > > > > - Reset the watermark to the initial watermark, which basically means > that > > instead of a "shutdown" tuple there is a "reset" tuple. > >

[jira] [Created] (APEXMALHAR-2492) Correct usage of empty Slice in Malhar Library

Ajay Gupta created APEXMALHAR-2492: -- Summary: Correct usage of empty Slice in Malhar Library Key: APEXMALHAR-2492 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2492 Project: Apache Apex

[jira] [Created] (APEXMALHAR-2491) GenericFileOutputOperatorTest : runTestApplication() does not call shutdown() and can continue to run.

Ajay Gupta created APEXMALHAR-2491: -- Summary: GenericFileOutputOperatorTest : runTestApplication() does not call shutdown() and can continue to run. Key: APEXMALHAR-2491 URL: https://issues.apache.org/jira

Re: Backward compatibility issue in 3.6.0 release

I would vote for 1 and making variables private since it anyways breaks semantic versioning. I think it would it be a good idea to release a 3.6.1 patch release as well. Ajay On Mon, May 15, 2017 at 10:36 PM, Sanjay Pujare wrote: > I vote for renaming to less common

[jira] [Created] (APEXMALHAR-2490) Change use of empty byte array to null while creating Slice object in TimeBasedDedupOperator

Ajay Gupta created APEXMALHAR-2490: -- Summary: Change use of empty byte array to null while creating Slice object in TimeBasedDedupOperator Key: APEXMALHAR-2490 URL: https://issues.apache.org/jira/browse

Proposal to upgrade Apex Core dependency in Malhar to 3.6

Hi all, Apex Malhar currently depends on core 3.4. Custom tuple support has been added to core in 3.6. Hence, malhar dependency needs to be updated. This is necessary for changing the malhar operators for batch use cases. Let me know the community view on the same. Thanks, Ajay

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

svParser = dag.addOperator("csvParser", CsvParser.class); > WindowedOperatorImpl<Pojo, Pojo, Pojo> windowedOperator = > dag.addOperator("windowOperator", WindowedOperatorImpl.class); > dag.addStream("csvToWindowed", csvParser.out, new > InputPort[]{wi

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

is not a good design decision, IMO. > > Thank you, > Vlad > > Отправлено с iPhone > > > On Apr 29, 2017, at 02:50, AJAY GUPTA <ajaygit...@gmail.com> wrote: > > > > I am using WindowedOperatorImpl and it is declared as follows. > > > > WindowedOper

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

Ajay On Fri, Apr 28, 2017 at 11:53 PM, Vlad Rozov <v.ro...@datatorrent.com> wrote: > How do you declare WindowedOperator? > > Thank you, > > Vlad > > > On 4/28/17 10:35, AJAY GUPTA wrote: > >> Vlad, >> >> The approach you suggested doesn't work

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

interface (it may > return itself in getValue()). > > Thank you, > > Vlad > > On 4/28/17 02:44, AJAY GUPTA wrote: > >> Hi All, >> >> I am creating an application which is using Windowed Operator. This >> application involves CsvParser operator

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

Hi All, I am creating an application which is using Windowed Operator. This application involves CsvParser operator emitting a POJO object which is to be passed as input to WindowedOperator. The WindowedOperator requires an instance of Tuple class as input : *public final transient

[jira] [Updated] (APEXMALHAR-2303) S3 Line By Line Module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Gupta updated APEXMALHAR-2303: --- Issue Type: New Feature (was: Bug) > S3 Line By Line Mod

Re: [VOTE] Apache Apex Malhar Release 3.7.0 (RC1)

In the release notes, the S3 Line by line module was in Bug. Should have been in new feature. I have updated the JIRA type to new feature. On Thu, Mar 30, 2017 at 9:49 AM, Pramod Immaneni wrote: > +1 binding > > Verified file and builder integrity. > Verified licenses. >

[jira] [Updated] (APEXMALHAR-2453) Add sort Accumulation for Windowed operator

[ https://issues.apache.org/jira/browse/APEXMALHAR-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Gupta updated APEXMALHAR-2453: --- Description: Currently there is no support for sorting data received as a part

[jira] [Commented] (APEXCORE-676) Show description for DefaultProperties only when user requests it

[ https://issues.apache.org/jira/browse/APEXCORE-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937136#comment-15937136 ] Ajay Gupta commented on APEXCORE-676: - Output without --with-description flag Command : get-app

[jira] [Assigned] (APEXCORE-676) Show description for DefaultProperties only when user requests it

[ https://issues.apache.org/jira/browse/APEXCORE-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Gupta reassigned APEXCORE-676: --- Assignee: Ajay Gupta > Show description for DefaultProperties only when user reque

[jira] [Assigned] (APEXMALHAR-2453) Sort Accumulation for Windowed operator

[ https://issues.apache.org/jira/browse/APEXMALHAR-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Gupta reassigned APEXMALHAR-2453: -- Assignee: Ajay Gupta > Sort Accumulation for Windowed opera

Re: Sort Accumulation

Hi Apex Dev community, Kindly let me know if implementing this accumulation using TreeMultiSet is fine. Ajay On Thu, Mar 9, 2017 at 12:24 PM, AJAY GUPTA <ajaygit...@gmail.com> wrote: > Hi Bright, > > I couldnot completely understand the bucketing approach you mentioned. How &g

Re: Watermark tuples in Apex

Hi Bhupesh, For point 1, cant we make use of implicitWatermarkGenerator? Ajay On Wed, Mar 8, 2017 at 12:16 PM, Bhupesh Chawda wrote: > Hi All, > > Watermark tuples in Apex are very tightly coupled to event time processing. > For this reason, usually they are modeled as

Re: Sort Accumulation

data with range of value. Such as following: > - process tuple in one window: sort data of current window in memory > - end window: merge the sorted memory data into buckets. > > thanks > Bright > > On Wed, Mar 8, 2017 at 8:51 AM, AJAY GUPTA <ajaygit...@gmail.com> wrote: > &

[Community Opinion] Use of Lombok Library

The Java operators we write have a lot of code which is less related to the business logic of the operator, eg. getters, setters, constructors, log variables, etc. Lombok library can be used to greatly avoid such code by using annotations, eg. @Getter, @Setter. The library generates this code

Re: PojoInnerJoin Accumulation emitting Map

Yes, it should be emitting a POJO. This POJO can then be further used to join with a third POJO stream, thus behaving like a DB join. It would be best if we can incorporate this into Schema Discovery design. Ajay On Wed, 15 Feb 2017 at 5:30 PM, Chinmay Kolhatkar wrote:

Re: APEXMALHAR-2400 In PojoInnerJoin accumulation same field names are emitted as single field

This is a bug if we consider a normal database join. All fields from both POJOs should be emitted in the result irrespective of the name. Ajay On Wed, 15 Feb 2017 at 5:55 PM, Hitesh Kapoor wrote: > Hi All, > > In PojoInnerJoin accumulation same field names are emitted

[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info

[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856498#comment-15856498 ] Ajay Gupta commented on APEXCORE-592: - [~sandesh] OperatorDiscoverer can give description, however

[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info

[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853604#comment-15853604 ] Ajay Gupta commented on APEXCORE-592: - Hi [~ashwinchandrap], considering the sope of this JIRA, I

Re: StrAM event grouping

+1 for the feature. This can in future be enhanced for other such groupings as well. Ajay On Wed, Jan 25, 2017 at 12:38 PM, Priyanka Gugale wrote: > Hi, > > StrAM raises events whenever Operator or Container restarts. In Apex Dag > whenever a container/operator, say

Re: One Yarn with Multiple Apex Applications

Hi Santhosh, We can definitely run multiple Apex applications on a single yarn instance. The behaviour in your case is most probably due to shortage of resources that were required by the second application. Once the first application was killed, the resources were released and the second

Re: [DISCUSS] Stram events log levels

, Jan 20, 2017 at 2:04 PM, AJAY GUPTA <ajaygit...@gmail.com> wrote: > Nice point Tushar. I will look into how we can pass on different event log > levels for same event type. > > > Ajay > > On Fri, Jan 20, 2017 at 10:18 AM, Tushar Gosavi <tus...@datatorrent.com>

Re: [DISCUSS] Stram events log levels

suggested by > Ajay > > doesn't seems to add significant overhead. Also I feel that in > future it > > can be enhanced further and UI can also filter the event logs based > on > > these log levels. > > > > On Jan 19, 2017 10:57 PM, "AJAY G

Re: [DISCUSS] Stram events log levels

, AJAY GUPTA <ajaygit...@gmail.com> wrote: > Hi Apex community, > > > Currently, events logged by stram such as StartContainer, StartOperator, > StopContainer, StopOperator, etc dont have any associated priority level. > > We can provide log levels for such Stram eve

[DISCUSS] Stram events log levels

Hi Apex community, Currently, events logged by stram such as StartContainer, StartOperator, StopContainer, StopOperator, etc dont have any associated priority level. We can provide log levels for such Stram events. Log levels can be INFO, WARN, ERROR Eg: 1. Start Container, Start Operator are

[jira] [Commented] (APEXCORE-611) Stram Event Log Levels

[ https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827493#comment-15827493 ] Ajay Gupta commented on APEXCORE-611: - Hi, The below approach can be used for this JIRA. Kindly let

[jira] [Created] (APEXCORE-611) Stram Event Log Levels

Ajay Gupta created APEXCORE-611: --- Summary: Stram Event Log Levels Key: APEXCORE-611 URL: https://issues.apache.org/jira/browse/APEXCORE-611 Project: Apache Apex Core Issue Type: Improvement

Re: Schema Discovery Support in Apex Applications

+1 for the idea. I just had one question. As I understand, there will be some form of Anonymous POJO used as objects to pass information from one operator to another. Can you share how the user/operator developer would access the tuple object in case he wishes to do something with it? Ajay On

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

_meta" > table is mandatory > for the operator to work correctly. Also provide a sample table creation > query for reference. > > On Sat, Jan 14, 2017 at 1:05 PM, AJAY GUPTA <ajaygit...@gmail.com> wrote: > > > Since the query can be different for differ

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

Since the query can be different for different databases, the user will have to provide query to the operator. Rather than this, I believe it's easier for user to directly execute create table query on DB. Also, the create table script won't be that heavy that we create script for it. Probably

[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info

[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818009#comment-15818009 ] Ajay Gupta commented on APEXCORE-592: - [~vrozov] I agree with your point that the description should

[jira] [Commented] (APEXCORE-601) provide log level in the event

[ https://issues.apache.org/jira/browse/APEXCORE-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811391#comment-15811391 ] Ajay Gupta commented on APEXCORE-601: - I can work on this > provide log level in the ev

[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info

[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804259#comment-15804259 ] Ajay Gupta commented on APEXCORE-592: - The output of get-app-package-info are the list of tags

[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info

[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763432#comment-15763432 ] Ajay Gupta commented on APEXCORE-592: - [~vrozov] The description in properties is required

Re: Shell script for launching and shutting down

Hi You can use the -e option to specify commands to be executed on apex cli For example : apex -e "list-apps" I had a single application running and following was the output apex -e "list-apps" {"apps": [{ "startTime": "2016-12-16 11:59:07 +0530", "id": 1, "name": "PiDemo", "state":

[jira] [Commented] (APEXCORE-561) Container info needs to be persisted even after application has been killed

[ https://issues.apache.org/jira/browse/APEXCORE-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717253#comment-15717253 ] Ajay Gupta commented on APEXCORE-561: - ContainerReport is already persisted in Yarn's timeline

Re: "ExcludeNodes" for an Apex application

Hi, Can't we make use of existing Node Label + queue feature in Yarn to achieve this. Though we will have to redeploy cluster, its still possible to exclude nodes. https://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/NodeLabel.html Thanks, Ajay On Fri, Dec 2, 2016 at 5:57 AM,

Re: Jenkins build failing

I see start point as origin/master and not origin/personal-branch. Can this be the issue? Ajay On Mon, 28 Nov 2016 at 4:28 PM, Hitesh Kapoor wrote: > Hi All, > > There seems to be an issue with Jenkins build which runs on creating a pull > request. > It is failing with

Re: Adding new log4j appender to Apex core

+1 for approach 2. Regards, Ajay On Wed, Nov 23, 2016 at 12:16 PM, David Yan wrote: > The goal of this log4j appender is to provide a log offset and the fixed > name of the container log file (instead of apex.log becoming apex.log.1 and > then apex.log.2, etc due to

Re: Proposal for apex/malhar extensions

+1 This is a good idea. Ajay On Wed, Nov 16, 2016 at 4:47 PM, Chinmay Kolhatkar wrote: > Dear Community, > > This is in relation to malhar cleanup work that is ongoing. > > In one of the talks during Apache BigData Europe, I got to know about > Spark-Packages

Re: [jira] [Commented] (APEXMALHAR-2303) S3 Line By Line Module

Few corrections in S3RecordReader > > > > > > S3 Line By Line Module > > -- > > > > Key: APEXMALHAR-2303 > > URL: https://issues.apache.org/ > jira/browse/APEXMALHAR-2303 > >

[jira] [Commented] (APEXMALHAR-2303) S3 Line By Line Module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614784#comment-15614784 ] Ajay Gupta commented on APEXMALHAR-2303: Raised PR for this code. Kindly review and let me

Re: [jira] [Assigned] (APEXMALHAR-2303) S3 Line By Line Module

other approaches you can think of. Thanks, Ajay On Wed, Oct 19, 2016 at 11:53 AM, AJAY GUPTA <ajaygit...@gmail.com> wrote: > Hi Apex Dev community, > > Kindly provide with feedback if any for the following approach for > implementing S3RecordReader. > > *S3R

Re: [jira] [Assigned] (APEXMALHAR-2303) S3 Line By Line Module

) as this set of bytes is a part of last record in previous block. Regards, Ajay On Wed, Oct 19, 2016 at 7:31 AM, Ajay Gupta (JIRA) <j...@apache.org> wrote: > > [ https://issues.apache.org/jira/browse/APEXMALHAR-2303? > page=com.atlassian.jira.plugin.system.issuetabpane

[jira] [Assigned] (APEXMALHAR-2303) S3 Line By Line Module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Gupta reassigned APEXMALHAR-2303: -- Assignee: Ajay Gupta > S3 Line By Line Mod

[jira] [Created] (APEXMALHAR-2303) S3 Line By Line Module

Ajay Gupta created APEXMALHAR-2303: -- Summary: S3 Line By Line Module Key: APEXMALHAR-2303 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2303 Project: Apache Apex Malhar Issue Type