apex on emr?

Hey guys, Any of you running Apex in production on EMR? -- Regards, Ashwin.

Re: Blocked operator PTOperator

Sunil, This might be related to checkpointing. See: https://github.com/apache/apex-core/blob/master/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java#L2211-L2217 Also check this piece of code:

Re: Occasional Out of order tuples when emitting from a thread

Sunil, You can poll the queue in end window since process method in the input port does not get called if there is no incoming tuple. However, end window is called irrespective of there are incoming tuples or not. Regards, Ashwin. On Tue, Feb 21, 2017 at 11:32 AM, Sunil Parmar

Re: One-time Initialization of in-memory data using a data file

Roger, Depending on the certain requirements on expected latency, size of data etc, the operator's design will change. If latency needs to be lowest possible, meaning completely in-memory and not hitting the disk for read I/O, there are two scenarios 1. If the lookup data size is small --> just

Re: PARTITION_PARALLEL Vs Regular mode

+1 for option 4 that Amol suggested. However, if you definitely need the shuffle, one thing you can check is if the key based partitioning is causing skews in the throughputs to individual parquet writer partitions. Regards, Ashwin. On Thu, Dec 22, 2016 at 1:22 PM, Amol Kekre

Re: Operators in Pending Deploy

l.com> wrote: > >> Yeah, application has been running for 20h. But no event is flowing >> through. See dt.log attached. >> >> On Fri, Dec 2, 2016 at 11:48 AM, Ashwin Chandra Putta < >> ashwinchand...@gmail.com> wrote: >> >>> Max, >>> >

Re: Operators in Pending Deploy

Max, Can you check app master logs? If application status changed to running, it means app master is running. You can find operator deployment related logs from app master dt.log. Regards, Ashwin. On Dec 2, 2016 5:25 AM, "Max Bridgewater" wrote: > I deployed the

Re: java.lang.NoClassDefFoundError: com/github/fge/jsonschema/exceptions/ProcessingException

Max, Seems like you need to add the following dependency. com.github.fge json-schema-validator 2.0.1 See the following example, but it is marked as optional so not sure if it is really required.

Re: Connection refused exception

Can you check in stram events on the gateway UI, you can select a time range to begin at the start of the application so you can see the series of events that happened in a timeline. Then you will find the first error in that list. You can find stram events in the logical tab to the right for the

Re: Apex Communication Protocols

Max, Apex deploys a fast pub-sub server called buffer server in every yarn container it gets (except AM) before deploying operators on it. For all the operators which are connected downstream to operators outside the current container, their output ports become publishers to the buffer server.

Re: Apex with SSH Tunel

gt; beyond virtual memory limits. Current usage: 532.4 MB of 1 GB physical > memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container. > > > I solved it by adding this to yarn-site.xml: > > > yarn.scheduler.minimum-allocation-mb > 1000 >

Re: Apex with SSH Tunel

ease find the AppMaster.stderr attached as well as dt.log. > AppMaster.stdout is empty. I am still wondering if there is another port > that is needed or if the UI is using websocket. > > On Sat, Nov 19, 2016 at 5:40 PM, Ashwin Chandra Putta < > ashwinchand...@gmail.com> wrote: > &g

Re: Apex with SSH Tunel

to FAILED. > > Max. > > On Fri, Nov 18, 2016 at 2:30 PM, Ashwin Chandra Putta < > ashwinchand...@gmail.com> wrote: > >> Also, there are write permission errors on /user/dtadmin/datatorrent in >> hdfs. Please make dtadmin user own /user/dtadmin/ >> &

Re: Apex with SSH Tunel

t; >> Here is the log folder. Note that it refers to a malformed >> properties.xml. I am attaching that properties file as well. >> >> On Fri, Nov 18, 2016 at 1:08 PM, Ashwin Chandra Putta < >> ashwinchand...@gmail.com> wrote: >> >>> Max, >>>

Re: Apex with SSH Tunel

Max, Can you share the gateway logs? You will find them under /var/log/datatorrent for global install, or under ~/.dt/logs for local install. Regards, Ashwin. Regards, Ashwin. On Nov 18, 2016 9:41 AM, "Max Bridgewater" wrote: > Hi Folks, > > I am playing with Apex

Re: Operators stay in PENDING_DEPLOY state

Dimple, Cluster does not have enough resources. Here is the initial container request set from the log. 2016-11-04 15:06:26,901 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[]Priority[0], Capability[

Re: How to use DataTorrent

Dimple, You don't need to use dtAssemble. You can use java to write the the applications. See example tutorials: https://github.com/DataTorrent/examples/tree/master/tutorials Regards, Ashwin. On Thu, Nov 3, 2016 at 4:01 PM, dimple wrote: > Is it necessary to use

Re: Retry functionality in Datatorrent

; > > Thanks!! > > On Wed, Oct 19, 2016 at 3:48 PM, Ashwin Chandra Putta < > ashwinchand...@gmail.com> wrote: > >> Ok, versions seem compatible. Can you do an ls on the target/deps/ >> directory and send the list of included jars? >> >> Regards, >>

Re: Retry functionality in Datatorrent

ent > *1.1.1-mapr-1602* > > > > > > Hadoop version : hadoop-2.7.0 (Mapr distribution 5.1) > > > > Thanks, > > Srinivas > > > > *From:* Ashwin Chandra Putta [mailto:ashwinchand...@gmail.com] > *Sent:* Wednesday, October 19, 2016 3:05

Re: Retry functionality in Datatorrent

wrote: > We have a Hbase client dependency and do not have any Hadoop dependencies > in project. > > > > *From:* Ashwin Chandra Putta [mailto:ashwinchand...@gmail.com] > *Sent:* Wednesday, October 19, 2016 2:40 PM > *To:* users@apex.apache.org > *Subject:* Re: R

Re: Retry functionality in Datatorrent

.java:589) > > at java.lang.Long.parseLong(Long.java:631) > > at org.apache.hadoop.yarn.util.ConverterUtils. > toApplicationAttemptId(ConverterUtils.java:137) > > at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId( > ConverterUtils.java:177) > > ... 1 more > > End

Re: balanced of Stream Codec

Sunil, For key based partitioning, the getPartition method is supposed to return a consistent integer representing the key for partitioning. Typically the java hashCode of the key. The tuples are then routed based on the integer and looking at its lower bits on the mask (number of lower bits)

Re: Datatorrent UI capabilities

Hey Srinivas, I will ask someone from DataTorrent to get in touch with you regarding the details. This forum is mainly for Apex related discussion which includes the open source Apex Core and Apex Malhar operator library. So DataTorrent UI discussion is usually outside the scope of this mailing

Re: dag.addStream Error

Is there only one KeyValue class in the classpath? Regards, Ashwin. On Mon, Aug 15, 2016 at 11:38 AM, Sanjay Pujare wrote: > This should work and I don’t see anything obviously wrong. > > > > Where is KeyValue defined and are you sure the same type is used > in

RE: Apex Finished Status

Doyle, Have you thrown it in all input operators? It starts from the operator you throw it in and propagates downstream. All operators need to shutdown before application changes state. Regards, Ashwin. On Aug 11, 2016 5:01 AM, "Doyle, Austin O." wrote: > The

Re: HDFS Write File Operator struggling to start

ication is running!! > > > Regards, > Raja. > > From: Ashwin Chandra Putta <ashwinchand...@gmail.com> > Reply-To: "users@apex.apache.org" <users@apex.apache.org> > Date: Thursday, August 4, 2016 at 4:42 PM > To: "users@apex.apache.org" <us

Re: Apache Apex and Dip blog

Created PR to add blog link: https://github.com/apache/apex-site/pull/46 Regards, Ashwin. On Mon, Aug 1, 2016 at 5:51 PM, Pramod Immaneni wrote: > Hey Neeraj, > > Nice blog. It would be great to link to it in the apex website blog > section in this page > >

Re: Sharing Properties Between Operators

Alex, I tried to give regular expression on the partial operator name and it works. Eg: dt.application.*.operator.hazel[\w].prop.host 127.0.0.1 Regards, Ashwin. On Mon, Aug 1, 2016 at 2:26 PM, David Yan wrote: > Hi Alex, > > You can make multiple operators

Re: Sub Partitioning the parallel partitions

Yogi, What is the reason for avoiding unifiers? Regards, Ashwin. On Mon, Jul 25, 2016 at 9:41 AM, Yogi Devendra wrote: > Hi, > > I have a DAG A->B->C. > > 1. A is kafka input operator reading from 4 different topics configured > with ONE_TO_ONE strategy. Thus creating

Re: Operators not working

When you say it does not emit anything, are you referring to your input operator? Is it an operator from malhar? Which one? And how many records do you have in the source table? Regards, Ashwin. On Jul 14, 2016 3:55 PM, "Jaikit Jilka" wrote: > Hello, > > I am parsing

Re: Runtime property modifications for an Apex Application

Raja, You can set log levels for class path patterns for running applications from gateway. Not sure if you can do it from CLI. http://docs.datatorrent.com/configuration/#adjusting-logging-levels Regards, Ashwin. On Tue, Jul 5, 2016 at 2:53 PM, Raja.Aravapalli