Re: example applications in malhar

2017-03-11 Thread Lakshmi Velineni 
Hi,

I have come up with the following list of applications to move from
datatorrent/examples into apex-malhar/examples. There were many examples
that do a similar functionality of moving data around from different
sources to destinations, so I separated them all into a separate folder
called "Ingestion".

csvformatter

dedup

dynamic-partition

enricher

exactly-once

filter

innerjoin

maprapp

parser

partition

recordreader

throttle

topnwords

transform

unifiers

*Ingestion/*

  cassandrainput

  cassandraoutput

  fileio

  fileio-multidir

  fileio-simple

  fileoutput

  filetojdbc

  hdfs2kafka

  hdfs-sync

  jdbcingest

  jdbctijdbc

  jmsActiveMQ

  jmsSqs

  kafka

  s3-to-hdfs-sync

  s3output


Let me know your thoughts.


Thanks

Lakshmi Prasanna







On Thu, Feb 23, 2017 at 10:34 AM, Amol Kekre  wrote:

> Yes, should merge samples into examples. Ideally the names of the examples
> should be more descriptive in terms of "how to" as opposed to a title. PI
> demo for example has lots of ways to do compute. So it could be named as
> "pi - distributed compute" in examples. Similarly if others can be named to
> bring out features to look at from the examples, it would make examples
> more useful to readers.
>
> Thks
> Amol
>
>
>
> E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>
> www.datatorrent.com  |  apex.apache.org
>
> *Join us at Apex Big Data World-San Jose
> , April 4, 2017!*
> [image: http://www.apexbigdata.com/san-jose-register.html]
> 
>
> On Thu, Feb 23, 2017 at 10:07 AM, Sanjay Pujare 
> wrote:
>
> > + for renaming to examples. While we are at it, how about merging
> "samples"
> > also in the new "examples" ?
> >
> > On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath 
> > wrote:
> >
> > > +1 for renaming to "examples"
> > >
> > > Ram
> > >
> > > On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <
> > laks...@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > I am ready to bring the examples over into the demos folder. I was
> > > > wondering if anybody has any input on Thomas's suggestion to rename
> the
> > > > demos folder to examples. I would rather do that first and then bring
> > the
> > > > examples over instead of doing it the other way around as that would
> > lead
> > > > to refactoring the new examples again.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> > > laks...@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Since the examples have little history I was planning to have two
> > > > > commits for every example, one for the code as the primary author
> of
> > > > > the example and another containing pom.xml and other changes to
> make
> > > > > it work under malhar.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > > >  wrote:
> > > > > > Thanks for the suggestions and I am working on the process to
> > migrate
> > > > the
> > > > > > examples with the guidelines you mentioned. I will send out a
> list
> > of
> > > > > > examples and the destination modules very soon.
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> > > thomas.we...@gmail.com>
> > > > > > wrote:
> > > > > >>
> > > > > >> Maybe a good first step would be to identify which examples to
> > bring
> > > > > over
> > > > > >> and where appropriate how to structure them in Malhar (for
> > example,
> > > I
> > > > > see
> > > > > >> multiple hdfs related apps that could go into the same Maven
> > > module).
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise 
> > > wrote:
> > > > > >>
> > > > > >> > That would be great. There are a few things to consider when
> > > working
> > > > > on
> > > > > >> > it:
> > > > > >> >
> > > > > >> > * preserve attribtion
> > > > > >> > * ensure there is a test that runs the application in the CI
> > > > > >> > * check that dependencies are compatible license
> > > > > >> > * maybe extract common boilerplate code from pom.xml
> > > > > >> >
> > > > > >> > etc.
> > > > > >> >
> > > > > >> > Existing examples are under https://github.com/apache/
> > > > > >> > apex-malhar/tree/master/demos
> > > > > >> >
> > > > > >> > Perhaps we should rename it to "examples"
> > > > > >> >
> > > > > >> > I also propose that each app has a README and we add those for
> > > > > existing
> > > > > >> > apps as well.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Thomas
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni <
> > > > > >> > laks...@datatorrent.com> wrote:
> > > > > >> >
> > > > > >> >>   Can i work on this?
> > > > > >> >>
> > > > > >> >> Thanks
> > > > > >> >> Lakshmi Prasanna
> > >

[jira] [Created] (APEXMALHAR-2443) User documentation for ManagedState and Spillable DS

2017-03-11 Thread Thomas Weise (JIRA) 
Thomas Weise created APEXMALHAR-2443:


 Summary: User documentation for ManagedState and Spillable DS
 Key: APEXMALHAR-2443
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2443
 Project: Apache Apex Malhar
  Issue Type: Documentation
Reporter: Thomas Weise


The component needs documentation for operator developers. It should explain 
what it does and how to use it to implement operators. Should include the 
assumptions about data access pattern, scalability and performance.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)