Please note that tuples should not be emitted by any thread other than the
main operator thread.
A common pattern is to use a thread-safe queue and have worker threads
enqueue
tuples there; the main operator thread then pulls tuples from the queue and
emits them.
Ram
On Tue, Feb 21, 2017 at 10:0
+1 for renaming to "examples"
Ram
On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni
wrote:
> I am ready to bring the examples over into the demos folder. I was
> wondering if anybody has any input on Thomas's suggestion to rename the
> demos folder to examples. I would rather do that first and
There was a response from Thomas 2 days ago which also included a code
fragment.
Let me know if you are unable to locate it and I'll copy it here.
Ram
On Fri, Mar 3, 2017 at 9:10 AM, Ganelin, Ilya
wrote:
> Hey all - any word on this? Would love to be able to test locally using
> the new framew
Getting: "Sorry, the file you have requested does not exist."
Ram
On Sun, Mar 26, 2017 at 11:18 PM, vikram patil
wrote:
> Hello All,
>
> I have developed python implementation allowing Apex Developers to create
> basic Apex Application using python. Still, there are some issues need to
> be re
Fixed the duplicate entries issue:
https://github.com/apache/apex-malhar/pull/592
Looking at the other one.
Ram
On Mon, Mar 27, 2017 at 11:58 PM, Thomas Weise wrote:
> The following two issues observed while preparing 3.7.0 RC1:
>
> User doc contains duplicate entries in left nav:
>
> http://a
The javadoc for 3.7 is there now -- I triggered a build via IRC.
Commit triggering was not working for me for some reason, so it is set to
build nightly.
Ram
On Mon, Mar 27, 2017 at 11:58 PM, Thomas Weise wrote:
> The following two issues observed while preparing 3.7.0 RC1:
>
> User doc contai
What's a use case for this ?
Ram
On Sat, Apr 1, 2017 at 8:12 AM, Vlad Rozov wrote:
> All,
>
> Currently Apex assumes that an operator can emit on any defined output
> port and all streams defined by a DAG are active. I'd like to propose an
> ability for an operator to open and close output port
People can currently do this by simply implementing the Appender interface
and adding it
with Logger.addAppender() in the setup method. Why do we need something
more elaborate ?
Ram
On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko
wrote:
> The configuration of a log4j appender via log4j configu
You can do it in a trivial derived class without changing the base class.
Ram
On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov
wrote:
> Does not the proposal to use Logger.addAppender() requires modifications
> to used operators code?
>
> Thank you,
>
> Vlad
>
> On 4/10/17
compiling applications. It is run-time
> configuration vs compile time hardcoded appender.
>
> Thank you,
>
> Vlad
> On 4/10/17 11:23, Munagala Ramanath wrote:
>
>> You can do it in a trivial derived class without changing the base class.
>>
>> Ram
>>
>
> > It will require application recompilation and repackaging. The proposed
> > functionality is for dev-ops to be able to route application logging to a
> > preferred destination without recompiling applications. It is run-time
> > configuration vs compile time hardcoded
It will be good to understand a use case where an operator needs a
> specific appender.
>
> IMO, an operator designer defines *what* should be logged and dev-ops team
> defines *where* to log.
>
> Thank you,
>
> Vlad
> On 4/10/17 16:27, Munagala Ramanath wrote:
>
an individual operators need to define a
> specific appender, can you provide one?
>
> Thank you,
>
> Vlad
>
> On 4/10/17 16:53, Munagala Ramanath wrote:
>
>> Yes, totally agree, it would be helpful to have a detailed use case and/or
>> a detailed spec
>> of the
I agree with Pramod that force pushing should be a rare event done only
when there is an immediate
need to undo something serious. Doing it just for a policy violation should
itself be codified in our
policies as a policy violation.
Why not just commit an improvement on top ?
Ram
On Thu, Apr 27,
ommits related to the plugin
> feature that IMO should be part of the original review. There wasn't any
> need to rush this, the change wasn't important for the release.
>
> Thomas
>
>
> On Thu, Apr 27, 2017 at 8:11 PM, Munagala Ramanath
> wrote:
>
> > I
we should not have to be back to the same topic a few days
> later.
>
> While you seem to be focussed on the disagreement on policy violation, I'm
> more interested in a style of collaboration that does not require such
> discussion.
>
> Thomas
>
> On Thu, Apr 27, 20
to be back to the same topic a few days
>> later.
>>
>> While you seem to be focussed on the disagreement on policy violation, I'm
>> more interested in a style of collaboration that does not require such
>> discussion.
>>
>> Thomas
>>
>>
The repos are OK now -- github has fixed the issue.
Ram
On Fri, Apr 28, 2017 at 2:44 PM, Pramod Immaneni
wrote:
> Could happen again so don't be alarmed.
>
> Thanks
>
> Begin forwarded message:
>
> *From:* Chris Lambertus
> *Date:* April 28, 2017 at 12:22:50 PM PDT
> *To:* committers
> *Subje
ers,
Jamie
On Fri, Apr 28, 2017 at 3:06 PM, Pramod Immaneni
wrote:
> The message below seems to indicate it is a temporary workaround and the
> fix isn't ready yet.
>
> Thanks
>
> On Fri, Apr 28, 2017 at 2:57 PM, Munagala Ramanath
> wrote:
>
> >
*Friday, April 28, 2017 at 10:09 AM
> >>> *To: *"dev@apex.apache.org"
> >>> *Subject: *Re: PR merge policy
> >>>
> >>>
> >>>
> >>>
> >>> On a lighter note, looks like the powers that be have been listen
alid reason to undo the commit (note that it
>>> is
>>> still committer responsibility to thoroughly review changes/new code and
>>> ensure that license is in place, build is free from compilation and other
>>> errors, code is optimal and readable, and basicall
gt;>>> merge their PR. While this might feel a little silly, it definitely
> >>>>> helps
> >>>>> to build collaboration, familiarity with the code base, and
> >>>>> intrinsically
> >>>>> avoids PRs being merged too
Now available at:
https://ci.apache.org/projects/apex-core/apex-core-javadoc-release-3.6/index.html
Ram
The injunction that tuple processing should be "as fast as possible" is based
on anassumption and a fact:
1. In most cases, users want to maximize application throughput.2. If a
callback (like beginWindow(), process(), endWindow(), etc.) takes too long,
the platform deems the operator hung and
I like proposal 1 too; I also agree with Ajay about doing a 2.6.1 patch release.
Ram
On Monday, May 15, 2017 10:18 AM, AJAY GUPTA wrote:
I would vote for 1 and making variables private since it anyways breaks
semantic versioning.
I think it would it be a good idea to release a 3.6.1 patc
+1 for option 2 (primary)
+1 for option 1 (secondary)
Ram
On Tuesday, August 22, 2017, 6:58:46 AM PDT, Vlad Rozov
wrote:
+1 for option 2 (primary)
+1 for option 1 (secondary)
Thank you,
Vlad
On 8/21/17 23:37, Ananth G wrote:
> +1 for option 2 and second vote for option 1
>
> Have we finaliz
Failing the CI build seems too drastic. My suggestion is to create a new
profile that activates thisfunctionality and encourage people to run with this
profile and file JIRAs when somethingexciting happens.
Ram
On Wednesday, November 1, 2017, 1:26:00 PM PDT, Thomas Weise
wrote:
Consid
+1
Ram
On Tuesday, December 19, 2017, 8:33:21 AM PST, Pramod Immaneni
wrote:
I have a mini proposal. The command get-app-package-info runs the
populateDAG method of an application to construct the DAG but does not
actually launch the DAG. An application developer does not know in which
Similarly, some links at http://apex.apache.org/docs.html are now broken (for
example http://docs.datatorrent.com/tutorials/topnwords/ and
http://docs.datatorrent.com/beginner/)
It would be useful if such content could be rehosted elsewhere.
Ram
On Thursday, May 3, 2018, 9:46:04 AM PDT, Thom
To me it seems a little too busy and intimidating to newcomers; I'd prefer
a simpler layout
not quite so minimalistic as the big G but, something like:
http://kafka.apache.org/
or http://geode.apache.org/ (though the 3x3 grid is still too much).
Ram
On Thu, Jun 23, 2016 at 10:08 AM, Amol Kekre w
What would the API look like for option 1 ? Another operator callback
called controlTuple() or does the operator code have to check each
incoming tuple to see if it was data or control ?
Ram
On Fri, Jun 24, 2016 at 11:42 PM, David Yan wrote:
> It looks like option 1 is preferred by the communit
Pramod, I'll take a look this evening if you can wait that long.
Ram
On Jul 5, 2016 11:24 AM, "Pramod Immaneni" wrote:
> I received some feedback. Any other comments before adding these guidelines
> to the project.
>
> Thanks
>
> On Fri, Jun 17, 2016 at 3:12 PM, Pramod Immaneni
> wrote:
>
> > Hi
We've had a number of issues recently related to dependencies on old
versions
of various packages/libraries such as Hadoop itself, Google guava,
HTTPClient,
mbassador, etc.
How about we create a "bleeding-edge" branch in both Core and Malhar which
will use the latest versions of these various depe
le
> > Hadoop 2.6 or later. Although some old versions of Cloudera that include
> > hadoop version earlier than 2.4 still have not reached end-of-life yet,
> the
> > number of users using those old versions is probably very small.
> >
> > David
> >
Could you share how you resolved the issue ?
Ram
On Mon, Jul 11, 2016 at 9:59 PM, Akshay Gore wrote:
> Thank you Pradeep for the response. I have already resolved the issue.
>
> Regards,
> Akshay
>
> On Tue, Jul 12, 2016 at 10:10 AM, Pradeep Kumbhar >
> wrote:
>
> > Should
> > "dt.application.
Anuj,
Could you provide additional details such as: What does your DAG look like
? What operators (custom as well as from Malhar)
are you using ? Does this exception happen immediately upon launch or after
some time ?
Ram
On Mon, Jul 11, 2016 at 10:12 PM, ANUJ THAKWANI
wrote:
> Hi,
>
> I have
+1
Ram
On Tue, Jul 12, 2016 at 2:35 PM, Pramod Immaneni
wrote:
> Hi,
>
> I was wondering how everyone felt about the volume of auto-generated emails
> on this list. Looks like multiple emails are generated and sent to everyone
> on the list even for relatively smaller actions such as commenting
There is already a link to a troubleshooting page at bottom of
https://apex.apache.org/docs.html
That page already has some discussion under the section entitled
"Calculating Container Memory"
so adding new content there seems like the right thing to do.
Ram
On Mon, Jul 18, 2016 at 11:27 PM, Chin
e location. I shall raise PR with
> the given suggestions.
>
> --prad
>
> On Tue, Jul 19, 2016 at 5:49 AM, Munagala Ramanath
> wrote:
>
> > There is already a link to a troubleshooting page at bottom of
> > https://apex.apache.org/docs.html
> > That page al
+1
Ram
On Thu, Aug 4, 2016 at 12:10 AM, Sandesh Hegde
wrote:
> Hello Team,
>
> This thread is to discuss the Named Checkpoint feature for Apex. (
> https://issues.apache.org/jira/browse/APEXCORE-498)
>
> Named checkpoints allow following workflow,
>
> 1. Users can trigger a checkpoint and give
I don't see any reason to allow either.
Ram
On Thu, Aug 4, 2016 at 8:51 AM, Vlad Rozov wrote:
> Currently addOperator/addStream/addModule allows both null and empty
> string in the operator/stream/module names. Is there any reason to allow
> empty string? Should empty string and null be disallo
eas.
>
>
> On 8/4/16, 9:48 AM, "Munagala Ramanath" wrote:
>
> I don't see any reason to allow either.
>
> Ram
>
> On Thu, Aug 4, 2016 at 8:51 AM, Vlad Rozov
> wrote:
>
> > Currently addOperator/addStream/addModule allows both
e your
> application fine with empty or null strings as names, is there any reason
> to make them mandatory?
>
> On the other hand, we can come up with a scheme for system generated names
> when the caller doesn’t provide a name. I have some ideas.
>
>
> On 8/4/16, 9:48
Ashwin Chandra Putta <
> > >>> ashwinchand...@gmail.com> wrote:
> > >>>
> > >>> > + dev@apex.apache.org
> > >>> > - us...@apex.apache.org
> > >>>
Just out of curiosity what is the smallest value that fixes the issue ?
Ram
On Thu, Aug 18, 2016 at 3:55 AM, Yogi Devendra
wrote:
> Hi,
>
> Current default value is 2000 ms.
>
> I tried an application which uses default value. It was noticed that it
> leads to continuous re-deployment of the op
A good start would be to revise the archetype to include as many
illustrative tests as reasonably possible -- people seem more willing to
follow examples than to follow instructions.
Ram
On Sep 12, 2016 5:26 PM, "Thomas Weise" wrote:
Hi,
Recently there was a bit of discussion on how to write te
Some info here:
http://docs.datatorrent.com/troubleshooting/#hadoop-dependencies-conflicts
Ram
On Wed, Sep 21, 2016 at 12:00 PM, Vlad Rozov
wrote:
> Is subject already documented?
>
> Thank you,
>
> Vlad
>
Definitely worth adding.
Ram
On Wed, Sep 21, 2016 at 1:20 PM, Pramod Immaneni
wrote:
> Candidate to be added here?
>
> https://apex.apache.org/docs/apex/development_best_practices/
>
> On Wed, Sep 21, 2016 at 12:24 PM, Munagala Ramanath
> wrote:
>
> &g
We've seen cases where operator state continues to grow without bound
either because
the developer was unaware of the importance of keeping state small or
because of some
anomaly downstream. In such cases, the operators could get killed with an
OOM exception because
these checkpoints are building
+1
Kudu looks impressive from the overview, though it seems to still be
maturing.
Ram
On Sat, Oct 1, 2016 at 11:42 PM, ananth wrote:
> Hello All,
>
> I was wondering if it would be worthwhile for the community to consider
> support for Apache Kudu as a store ( as a contrib operator inside Apa
+1 for 2.6.x
Ram
On Mon, Oct 3, 2016 at 1:47 PM, David Yan wrote:
> Hi all,
>
> Thomas created this ticket for upgrading our Hadoop dependency version a
> couple weeks ago:
>
> https://issues.apache.org/jira/browse/APEXCORE-536
>
> We'd like to get the ball rolling and would like to take a vote
Could Connection.getTransactionIsolation() be used to determine if
transactions are supported ?
Ram
On Mon, Oct 24, 2016 at 8:08 AM, Hitesh Kapoor
wrote:
> Hi All,
>
> Need your views and suggestions regarding JdbcPOJOInsertOutput operator.
> This operator creates a transaction at the start of
Try changing *setFieldInfo* to* setFieldInfoItem*
Ram
On Mon, Nov 7, 2016 at 4:23 AM, Hitesh Kapoor
wrote:
> Hi All,
>
>
> Currently in JdbcPOJOInsertOuput operator we cannot configure JdbcFieldInfo
> via properties.xml and the user has to do the necessary coding in his
> application.
> To star
When would the visits happen ? Just before normal validation ?
Ram
On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi wrote:
> Hi All,
>
> How about adding visitor like API for DAG in Apex, and an api to
> register visitor for the DAG.
> Possible use cases are
> - Validator visitor which could val
of
> scenerios.
>
> Thks
> Amol
>
> On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi >
> wrote:
>
> > Yes, It could happen after current DAG validation and before the
> > application master is launched.
> >
> >
But then, what's the solution to the 2 problem scenarios that Milind
describes ?
Ram
On Wed, Nov 30, 2016 at 10:34 AM, Sanjay Pujare
wrote:
> I think “exclude nodes” and such is really the job of the resource manager
> i.e. Yarn. So I am not sure taking over some of these tasks in Apex would
>
ache.org/jira/browse/YARN-1042) so if Apex needs it we
> will need to do it ourselves.
>
> On 11/30/16, 10:39 AM, "Munagala Ramanath" wrote:
>
> But then, what's the solution to the 2 problem scenarios that Milind
> describes ?
>
> Ram
>
>
Bright,
We have this in contrib/pom.xml which I think is causing the problem:
com.google.guava
guava
16.0.1
provided
true
whereas the hadoop-common 2.2.0 depends on 11.0.2
Ram
On Thu, Dec 1, 2016 at 4:11 PM, Bright Chen wrote:
> Hi All,
> I got an runt
It only takes effect after failures -- no way to exclude from the get-go.
Ram
On Dec 1, 2016 7:15 PM, "Bhupesh Chawda" wrote:
> As suggested by Sandesh, the parameter
> MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST seems to do exactly what
> is needed.
> Why would this not work?
>
> ~ Bhupes
g anti-affinity.
> > >>
> > >> On Fri, Dec 2, 2016 at 10:03 AM, Bhupesh Chawda <
> > bhup...@datatorrent.com>
> > >> wrote:
> > >>
> > >>> Okay, I think that serves an alternate purpose of detecting any ne
r off clearly articulating and allowing users to
> > > >> configure it seperately as against using anti-affinity.
> > > >>
> > > >> On Fri, Dec 2, 2016 at 10:03 AM, Bhupesh Chawda <
> > > bhup...@datatorrent.com>
> > > >>
, Dec 2, 2016 at 7:13 AM, Munagala Ramanath
> wrote:
>
> > Could STRAM include a poison pill where it simply exits with diagnostic
> if
> > its host name is blacklisted ?
> >
> > Ram
> >
> > On Thu, Dec 1, 2016 at 11:52 PM, Amol Kekre
> wrote:
> >
>
In Netlet CircularBuffer constructor, we have an infinite loop if the first
parameter (*n*)
is *Integer.MAX_VALUE* because the loop counter left-shifts 1 till it drops
into the sign
bit at which point the value is negative and fails the loop exit test. The
next left shift
yields 0 which, of course,
Actually, this will happen whenever the parameter n satisfies: 2**30 < n <=
Integer.MAX_VALUE
Ram
On Thu, Dec 22, 2016 at 5:34 PM, Munagala Ramanath
wrote:
> In Netlet CircularBuffer constructor, we have an infinite loop if the
> first parameter (*n*)
> is *Integer.MAX_VALUE* b
Yes, I think an initial discussion that includes some or all of the
following
would be invaluable both for feature implementations and for bug fixes:
1. Discussion of current implementation and how/why it fails to meet
current need.
2. Possible approaches and tradeoffs (if any) for each.
3. Recomme
Are you running on the sandbox (if so, what version ?) or your own cluster ?
In either case, please check the following configuration item in
capacity-scheduler.xml:
yarn.scheduler.capacity.maximum-am-resource-percent
0.1
Maximum percent of resources in the cluster which can
Maybe they are too big and being stripped out by the mail programs ?
'cause I'm still not seeing them.
Ram
On Fri, Feb 3, 2017 at 12:15 PM, Michelle Xiao
wrote:
> Here are the screenshots.
>
>
> Thanks,
> Michelle
>
>
>
> On Fri, Feb 3, 2017 at 11:56 AM, Pramod Immaneni
> wrote:
>
>> Michelle,
There is some discussion of Kryo serializer issues at:
http://docs.datatorrent.com/troubleshooting/
under the heading "Application throwing following Kryo exception."
Ram
On Tue, Feb 7, 2017 at 12:22 AM, Ambarish Pande <
ambarish.pande2...@gmail.com> wrote:
> I understand that it affects perform
;
> Thank you,
>
> Vlad
>
> On 12/22/16 18:02, Munagala Ramanath wrote:
>
>> Actually, this will happen whenever the parameter n satisfies: 2**30 < n
>> <=
>> Integer.MAX_VALUE
>>
>> Ram
>>
>> On Thu, Dec 22, 2016 at 5:34 PM, Munagala
You can't do it in populateDAG since the id is not known when that function
runs.
You need to do it in the operator as suggested by Gaurav:
String appid = Context.OperatorContext.getValue(Context.DAGContext.
APPLICATION_ID);
Ram
On Wed, May 25, 2016 at 11:00 AM, Ganelin, Ilya wrote:
> Hi guys
Anybody know if there are plans for a Parquet writer operator ? If so, can
anyone
share status and timeline ?
Thanks.
Ram
+1
Since we have *setInputPortAttribute* and *setOutputPortAttribute*, it
seems reasonable
to add *setOperatorAttribute*.
Ram
On Mon, Jun 6, 2016 at 1:39 PM, Sandesh Hegde
wrote:
> Currently, *setAttribute* is used to set the operator attributes. Other 2
> Attribute setting APIs are specific t
Tim,
Are you building a config package by any chance, since that builds an *apc*
file.
Can you post the exact archetype command you ran ?
I just ran the archetype command for 3.4.0 and 3.5.0-SNAPSHOT and it built
an *apa* just as it always does.
Ram
On Tue, Jun 7, 2016 at 12:44 PM, Timothy Fark
gt;
> Thanks,
> Tim
>
> On Tue, Jun 7, 2016 at 1:15 PM, Munagala Ramanath
> wrote:
>
> > Tim,
> >
> > Are you building a config package by any chance, since that builds an
> *apc*
> > file.
> > Can you post the exact archetype command you ran ?
>
Also, last I looked, the sandbox was still 3.3.1
Ram
On Tue, Jun 7, 2016 at 1:29 PM, Timothy Farkas wrote:
> Thanks Ram, will take a note.
>
> Tim
>
> On Tue, Jun 7, 2016 at 1:27 PM, Munagala Ramanath
> wrote:
>
> > The archetype command for each version is in the f
Seems like a good idea.
Make sure that, with the new way, we can still set properties of Unifiers
too -- I saw some
special case code for that, don't recall details.
Ram
On Thu, Jun 9, 2016 at 9:04 AM, Sandesh Hegde
wrote:
> There is a parallel between StreamCodec and Unifier. As StreamCodec c
76 matches
Mail list logo
