Re: [ANNOUNCE] New Committer: Sandeep Samudrala
Congratulations Sandeep. Well deserved! On Thu, Mar 9, 2017 at 3:53 AM Praveen Adlakha <praveen.adla...@inmobi.com> wrote: > Congratulations Sandeep! > > On Thu, Mar 9, 2017 at 12:46 PM, pragya mittal <mittal.pragy...@gmail.com> > wrote: > > > Congratulations Sandeep... Well deserved :) > > > > On Thu, Mar 9, 2017 at 12:00 PM, pavan kumar Kolamuri < > > pavan.kolam...@gmail.com> wrote: > > > > > Congrats Sandeep. Very well deserved :) > > > > > > On Thu, Mar 9, 2017 at 11:40 AM, Venkat Ranganathan < > > > vranganat...@hortonworks.com> wrote: > > > > > > > Congratulations Sandeep! > > > > > > > > Venkat > > > > > > > > On 3/8/17, 9:25 PM, "Pallavi Rao" <pallavi@inmobi.com> wrote: > > > > > > > > The Project Management Committee (PMC) of Apache Falcon has asked > > > > Sandeep > > > > Samudrala to become a committer and we are pleased to announce > that > > > he > > > > has > > > > accepted. > > > > > > > > Regards > > > > Pallavi Rao (on behalf of Apache Falcon PMC) > > > > > > > > -- > > > > _ > > > > The information contained in this communication is intended > solely > > > for > > > > the > > > > use of the individual or entity to whom it is addressed and > others > > > > authorized to receive it. It may contain confidential or legally > > > > privileged > > > > information. If you are not the intended recipient you are hereby > > > > notified > > > > that any disclosure, copying, distribution or taking any action > in > > > > reliance > > > > on the contents of this information is strictly prohibited and > may > > be > > > > unlawful. If you have received this communication in error, > please > > > > notify > > > > us immediately by responding to this email and then delete it > from > > > your > > > > system. The firm is neither liable for the proper and complete > > > > transmission > > > > of the information contained in this communication nor for any > > delay > > > > in its > > > > receipt. > > > > > > > > > > > > > > > > > > > > > -- > > > Regards > > > Pavan Kumar Kolamuri > > > > > > > -- > _ > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you have received this communication in error, please notify > us immediately by responding to this email and then delete it from your > system. The firm is neither liable for the proper and complete transmission > of the information contained in this communication nor for any delay in its > receipt. > -- Regards Ajay Yadava
[jira] [Updated] (FALCON-436) Addition of a feed monitoring service in Falcon to report the feed SLAs breach
[ https://issues.apache.org/jira/browse/FALCON-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-436: --- Assignee: (was: Ajay Yadava) > Addition of a feed monitoring service in Falcon to report the feed SLAs breach > -- > > Key: FALCON-436 > URL: https://issues.apache.org/jira/browse/FALCON-436 > Project: Falcon > Issue Type: New Feature >Reporter: Rohit Kochar > > I have a use case where i want to monitor whether a particular instance of a > feed was published before a pre defined time and report this metrics to some > central service. > I believe this can be achieved by defining a SLA parameter of each feed and > by addition of feed monitoring service in the Falcon which monitors the feed > instances. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Unable to submit cluster definition
Great! Glad to hear it.
On Wed, Jan 11, 2017 at 11:29 AM Aaron Turner <synfina...@gmail.com> wrote:
> Hi Ajay,
>
> Made a lot of progress last night thanks to Sandeep and Srikanth(??) .
> Have a few more things to take care of which will hopefully be the
> last. I'll follow up one way or another.
>
> Thanks,
> Aaron
>
> --
> Aaron Turner
> https://synfin.net/ Twitter: @synfinatic
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
>
>
> On Wed, Jan 11, 2017 at 7:54 AM, Ajay Yadava <ajayyad...@apache.org>
> wrote:
> > Hi Aaron,
> >
> > Was your issue resolved?
> >
> > On Tue, Jan 10, 2017 at 11:21 PM Aaron Turner <synfina...@gmail.com>
> wrote:
> >
> > Just figured I'd let everyone know I'm on the IRC channel.
> > --
> > Aaron Turner
> > https://synfin.net/ Twitter: @synfinatic
> > Those who would give up essential Liberty, to purchase a little temporary
> > Safety, deserve neither Liberty nor Safety.
> > -- Benjamin Franklin
> >
> >
> > On Tue, Jan 10, 2017 at 4:46 PM, Aaron Turner <synfina...@gmail.com>
> wrote:
> >> Thanks for the clarification on the attachments. Yes, the properties
> >> files are currently the same file (our config is managed by puppet).
> >>
> >> Sadly, our use case requires distributed mode.
> >> --
> >> Aaron Turner
> >> https://synfin.net/ Twitter: @synfinatic
> >> Those who would give up essential Liberty, to purchase a little
> temporary
> >> Safety, deserve neither Liberty nor Safety.
> >> -- Benjamin Franklin
> >>
> >>
> >> On Tue, Jan 10, 2017 at 1:31 PM, Ajay Yadava <ajayyad...@apache.org>
> > wrote:
> >>> {quote}
> >>> ... not sure if the list removed them from my email?
> >>> {quote}
> >>> Yes, attachments are stripped on Apache mailing lists.
> >>>
> >>> Are you using the same startup and runtime.properties for both
> >>> *falcon-server* and the *falcon-prism*?
> >>>
> >>> Also, if it fits your use case, you may also want to look at the
> embedded
> >>> mode, which is the production equivalent of distributed mode for single
> >>> data center use cases. It is definitely much easier to understand/debug
> >>> etc. and you don't have to run multiple instances of falcon-prism and
> >>> falcon-server.
> >>>
> >>> Cheers
> >>> Ajay Yadava
> >>>
> >>> On Tue, Jan 10, 2017 at 11:58 AM Aaron Turner <synfina...@gmail.com>
> > wrote:
> >>>
> >>>> So gmail shows I attached the files... not sure if the list removed
> >>>> them from my email? Anyways, I"m GMT-8, so my evening/your morning
> >>>> would be best.
> >>>>
> >>>> Anyways, I've posted the configs to pastebin:
> >>>> http://pastebin.com/h5M6Mj06
> >>>> http://pastebin.com/PjP63qeE
> >>>> http://pastebin.com/d5S4KDTH
> >>>>
> >>>> --
> >>>> Aaron Turner
> >>>> https://synfin.net/ Twitter: @synfinatic
> >>>> Those who would give up essential Liberty, to purchase a little
> > temporary
> >>>> Safety, deserve neither Liberty nor Safety.
> >>>> -- Benjamin Franklin
> >>>>
> >>>>
> >>>> On Tue, Jan 10, 2017 at 4:15 AM, Srikanth Sundarrajan
> >>>> <srik...@hotmail.com> wrote:
> >>>> > Hi Aaron,
> >>>> >
> >>>> > Please let us know a tentative time (along with TZ) that you
> will
> > be
> >>>> around, we can try and login around the same time.
> >>>> >
> >>>> > Regards
> >>>> > Srikanth Sundarrajan
> >>>> >
> >>>> > From: Aaron Turner <synfina...@gmail.com>
> >>>> > Sent: Tuesday, January 10, 2017 8:49 AM
> >>>> > To: dev@falcon.apache.org
> >>>> > Subject: Re: Unable to submit cluster definition
> >>>> >
> >>>> > What time are people usually on the IRC channel? I've tried
> >>>> > connecting a few times and nobody else is there. :(
> >>>> > --
> >>
Re: Unable to submit cluster definition
{quote}
... not sure if the list removed them from my email?
{quote}
Yes, attachments are stripped on Apache mailing lists.
Are you using the same startup and runtime.properties for both
*falcon-server* and the *falcon-prism*?
Also, if it fits your use case, you may also want to look at the embedded
mode, which is the production equivalent of distributed mode for single
data center use cases. It is definitely much easier to understand/debug
etc. and you don't have to run multiple instances of falcon-prism and
falcon-server.
Cheers
Ajay Yadava
On Tue, Jan 10, 2017 at 11:58 AM Aaron Turner <synfina...@gmail.com> wrote:
> So gmail shows I attached the files... not sure if the list removed
> them from my email? Anyways, I"m GMT-8, so my evening/your morning
> would be best.
>
> Anyways, I've posted the configs to pastebin:
> http://pastebin.com/h5M6Mj06
> http://pastebin.com/PjP63qeE
> http://pastebin.com/d5S4KDTH
>
> --
> Aaron Turner
> https://synfin.net/ Twitter: @synfinatic
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
>
>
> On Tue, Jan 10, 2017 at 4:15 AM, Srikanth Sundarrajan
> <srik...@hotmail.com> wrote:
> > Hi Aaron,
> >
> > Please let us know a tentative time (along with TZ) that you will be
> around, we can try and login around the same time.
> >
> > Regards
> > Srikanth Sundarrajan
> >
> > From: Aaron Turner <synfina...@gmail.com>
> > Sent: Tuesday, January 10, 2017 8:49 AM
> > To: dev@falcon.apache.org
> > Subject: Re: Unable to submit cluster definition
> >
> > What time are people usually on the IRC channel? I've tried
> > connecting a few times and nobody else is there. :(
> > --
> > Aaron Turner
> > https://synfin.net/ Twitter: @synfinatic
> > Syn Fin dot Net | Streaming Thoughts from Syn to Fin<https://synfin.net/
> >
> > synfin.net
> > It's been a few years, but I really wanted to do a write up how I came
> to build my race bike: a Suzuki SV650 powered Ducati 1098S- or as I like to
> call it, a ...
> >
> >
> >
> > Those who would give up essential Liberty, to purchase a little temporary
> > Safety, deserve neither Liberty nor Safety.
> > -- Benjamin Franklin
> >
> >
> > On Mon, Jan 9, 2017 at 6:13 PM, Srikanth Sundarrajan
> > <srik...@hotmail.com> wrote:
> >> Would you want to join on the IRC, one or more of us can be on the IRC
> to help debug the issue along with you. We can bring back the findings and
> resolution to the JIRA post that.
> >>
> >>
> >> http://webchat.freenode.net/?channels=apachefalcon=d4
> > freenode Web IRC (qwebirc)<
> http://webchat.freenode.net/?channels=apachefalcon=d4>
> > webchat.freenode.net
> > Javascript is required to use IRC.
> >
> >
> >
> >>
> >>
> >> Regards
> >>
> >> Srikanth Sundarrajan
> >>
> >> From: Aaron Turner <synfina...@gmail.com>
> >> Sent: Monday, January 9, 2017 10:05 PM
> >> To: dev@falcon.apache.org
> >> Subject: Re: Unable to submit cluster definition
> >>
> >> On Fri, Jan 6, 2017 at 8:18 PM, Sandeep Samudrala <sandys...@gmail.com>
> wrote:
> >>> Please add "falcon.current.colo=eng" to both runtime.properties in
> prism
> >>> and falcon and then try it out.
> >>
> >> So I changed *.current.colo=eng to falcon.current.colo=eng and
> >> restarted server/prism. I'm getting the exact same error in the
> >> falcon CLI tool and the server/prism logs.
> >>
> >> --
> >> Aaron Turner
> >> https://synfin.net/ Twitter: @synfinatic
> >> Syn Fin dot Net | Streaming Thoughts from Syn to Fin<
> https://synfin.net/>
> >> synfin.net
> >> It's been a few years, but I really wanted to do a write up how I came
> to build my race bike: a Suzuki SV650 powered Ducati 1098S- or as I like to
> call it, a ...
> >>
> >>
> >>
> >> Those who would give up essential Liberty, to purchase a little
> temporary
> >> Safety, deserve neither Liberty nor Safety.
> >> -- Benjamin Franklin
>
--
Regards
Ajay Yadava
[jira] [Created] (FALCON-2244) Add users mailing list to the list of mailing lists on the website
Ajay Yadava created FALCON-2244: --- Summary: Add users mailing list to the list of mailing lists on the website Key: FALCON-2244 URL: https://issues.apache.org/jira/browse/FALCON-2244 Project: Falcon Issue Type: Improvement Reporter: Ajay Yadava Assignee: Ajay Yadava We have a users mailing list but it is not advertised on the website along with other mailing lists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FALCON-2234) Falcon feed fails for RDBMS import if extract type is incremental
[
https://issues.apache.org/jira/browse/FALCON-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813741#comment-15813741
]
Ajay Yadava edited comment on FALCON-2234 at 1/10/17 3:39 AM:
--
Yes, I think until the backend also supports it, we should disable that option
from UI and update the documentation to reflect the same. Otherwise, it is a
source of great confusion.
Copying Venkat and Venky for a second pair of eyes on this.
CC [~venkatnrangan] [~venky]
was (Author: ajayyadava):
Yes, I think until the backend also supports it, we should disable that option
from UI and update the documentation to reflect the same. Otherwise, it is a
source of great confusion.
CC [~venkatnrangan] [~venky]
> Falcon feed fails for RDBMS import if extract type is incremental
> -
>
> Key: FALCON-2234
> URL: https://issues.apache.org/jira/browse/FALCON-2234
> Project: Falcon
> Issue Type: Bug
> Components: falcon-ui
>Affects Versions: 0.10
>Reporter: Prashant
>Priority: Critical
> Labels: documentaion, newbie
>
> Falcon feed is failing with
> org.apache.falcon.entity.parser.ValidationException
> when extract type is incremental.
> As per the Apache doc Falcon supports
> Import policy
> The valid combinations are:
> [full,snapshot] - data is extracted in full and dumped into the feed instance
> location.
> [incremental, append] - data is extracted incrementally using the key
> specified in the deltacolumn
> As per the below code , it appears we only check [full,snapshot] and
> [incremental, append] goes to exception
> https://github.com/apache/falcon/blob/0.10/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
> Snippet
> {code}
> /**
> * Validate extraction and merge type combination. Currently supported
> combo:
> *
> * ExtractionType = FULL and MergeType = SNAPSHOT.
> * ExtractionType = INCREMENTAL and MergeType = APPEND.
> *
> * @param feed Feed entity
> * @param cluster Cluster referenced in the Feed definition
> * @throws FalconException
> */
> private void validateFeedExtractionType(Feed feed, Cluster cluster)
> throws FalconException {
> Extract extract = cluster.getImport().getSource().getExtract();
> if (ExtractMethod.FULL == extract.getType()) {
> if ((MergeType.SNAPSHOT != extract.getMergepolicy())
> || (extract.getDeltacolumn() != null)) {
> throw new ValidationException(String.format("Feed %s is using
> FULL "
> + "extract method but specifies either a superfluous "
> + "deltacolumn or a mergepolicy other than
> snapshot", feed.getName()));
> }
> } else {
> throw new ValidationException(String.format("Feed %s is using
> unsupported "
> + "extraction mechanism %s", feed.getName(),
> extract.getType().value()));
> }
> }
> {code}
> It must have a clause to also check [incremental, append] .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FALCON-2234) Falcon feed fails for RDBMS import if extract type is incremental
[
https://issues.apache.org/jira/browse/FALCON-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813741#comment-15813741
]
Ajay Yadava commented on FALCON-2234:
-
Yes, I think until the backend also supports it, we should disable that option
from UI and update the documentation to reflect the same. Otherwise, it is a
source of great confusion.
CC [~venkatnrangan] [~venky]
> Falcon feed fails for RDBMS import if extract type is incremental
> -
>
> Key: FALCON-2234
> URL: https://issues.apache.org/jira/browse/FALCON-2234
> Project: Falcon
> Issue Type: Bug
> Components: falcon-ui
>Affects Versions: 0.10
>Reporter: Prashant
>Priority: Critical
> Labels: documentaion, newbie
>
> Falcon feed is failing with
> org.apache.falcon.entity.parser.ValidationException
> when extract type is incremental.
> As per the Apache doc Falcon supports
> Import policy
> The valid combinations are:
> [full,snapshot] - data is extracted in full and dumped into the feed instance
> location.
> [incremental, append] - data is extracted incrementally using the key
> specified in the deltacolumn
> As per the below code , it appears we only check [full,snapshot] and
> [incremental, append] goes to exception
> https://github.com/apache/falcon/blob/0.10/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
> Snippet
> {code}
> /**
> * Validate extraction and merge type combination. Currently supported
> combo:
> *
> * ExtractionType = FULL and MergeType = SNAPSHOT.
> * ExtractionType = INCREMENTAL and MergeType = APPEND.
> *
> * @param feed Feed entity
> * @param cluster Cluster referenced in the Feed definition
> * @throws FalconException
> */
> private void validateFeedExtractionType(Feed feed, Cluster cluster)
> throws FalconException {
> Extract extract = cluster.getImport().getSource().getExtract();
> if (ExtractMethod.FULL == extract.getType()) {
> if ((MergeType.SNAPSHOT != extract.getMergepolicy())
> || (extract.getDeltacolumn() != null)) {
> throw new ValidationException(String.format("Feed %s is using
> FULL "
> + "extract method but specifies either a superfluous "
> + "deltacolumn or a mergepolicy other than
> snapshot", feed.getName()));
> }
> } else {
> throw new ValidationException(String.format("Feed %s is using
> unsupported "
> + "extraction mechanism %s", feed.getName(),
> extract.getType().value()));
> }
> }
> {code}
> It must have a clause to also check [incremental, append] .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FALCON-1872) Count Based Retention Policies
[ https://issues.apache.org/jira/browse/FALCON-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805803#comment-15805803 ] Ajay Yadava commented on FALCON-1872: - Assigning to [~pracheer] as he is working on this now. This is a more generic version(possible duplicate) of FALCON-705. > Count Based Retention Policies > -- > > Key: FALCON-1872 > URL: https://issues.apache.org/jira/browse/FALCON-1872 > Project: Falcon > Issue Type: New Feature > Components: feed-lifecycle > Reporter: Ajay Yadava >Assignee: Pracheer Agarwal > Labels: lifecycle > Fix For: 1.0 > > > Current retention policy supports deletion based only on nominal time. We > should have count based retention policies. Count should be determined on the > basis of instance time / access time / nominal time as configured by user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-1872) Count Based Retention Policies
[ https://issues.apache.org/jira/browse/FALCON-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-1872: Assignee: Pracheer Agarwal (was: Ajay Yadava) > Count Based Retention Policies > -- > > Key: FALCON-1872 > URL: https://issues.apache.org/jira/browse/FALCON-1872 > Project: Falcon > Issue Type: New Feature > Components: feed-lifecycle > Reporter: Ajay Yadava >Assignee: Pracheer Agarwal > Labels: lifecycle > Fix For: 1.0 > > > Current retention policy supports deletion based only on nominal time. We > should have count based retention policies. Count should be determined on the > basis of instance time / access time / nominal time as configured by user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-1851) Enable Replication as a lifecycle stage
[ https://issues.apache.org/jira/browse/FALCON-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804751#comment-15804751 ] Ajay Yadava commented on FALCON-1851: - You have taken up quite an impressive task! Thank you for taking this up. May I recommend, a design doc as the first step? You may also want to implement the other lifecycle task, implementing a new policy for retention on the basis of the number of instances, as a stepping stone to this one. > Enable Replication as a lifecycle stage > --- > > Key: FALCON-1851 > URL: https://issues.apache.org/jira/browse/FALCON-1851 > Project: Falcon > Issue Type: New Feature > Reporter: Ajay Yadava >Assignee: Pracheer Agarwal > Labels: gsoc2016, mentor > > As part of FALCON-965 we introduced a new way of specifying different stages > of a feed. As part of FALCON-965 we moved retention to lifecycle framework. > As part of our plan to migrate completely towards lifecycle framework, we > would want to allow users to specify replication as a lifecycle stage as > well. > For more details please refer to [Wiki Page | > https://cwiki.apache.org/confluence/display/FALCON/Replication+as+a+lifecycle+stage] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2234) Falcon feed fails for RDBMS import if extract type is incremental
[
https://issues.apache.org/jira/browse/FALCON-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Yadava updated FALCON-2234:
Labels: documentaion newbie (was: )
> Falcon feed fails for RDBMS import if extract type is incremental
> -
>
> Key: FALCON-2234
> URL: https://issues.apache.org/jira/browse/FALCON-2234
> Project: Falcon
> Issue Type: Bug
> Components: falcon-ui
>Affects Versions: 0.10
>Reporter: Prashant
>Priority: Critical
> Labels: documentaion, newbie
>
> Falcon feed is failing with
> org.apache.falcon.entity.parser.ValidationException
> when extract type is incremental.
> As per the Apache doc Falcon supports
> Import policy
> The valid combinations are:
> [full,snapshot] - data is extracted in full and dumped into the feed instance
> location.
> [incremental, append] - data is extracted incrementally using the key
> specified in the deltacolumn
> As per the below code , it appears we only check [full,snapshot] and
> [incremental, append] goes to exception
> https://github.com/apache/falcon/blob/0.10/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
> Snippet
> {code}
> /**
> * Validate extraction and merge type combination. Currently supported
> combo:
> *
> * ExtractionType = FULL and MergeType = SNAPSHOT.
> * ExtractionType = INCREMENTAL and MergeType = APPEND.
> *
> * @param feed Feed entity
> * @param cluster Cluster referenced in the Feed definition
> * @throws FalconException
> */
> private void validateFeedExtractionType(Feed feed, Cluster cluster)
> throws FalconException {
> Extract extract = cluster.getImport().getSource().getExtract();
> if (ExtractMethod.FULL == extract.getType()) {
> if ((MergeType.SNAPSHOT != extract.getMergepolicy())
> || (extract.getDeltacolumn() != null)) {
> throw new ValidationException(String.format("Feed %s is using
> FULL "
> + "extract method but specifies either a superfluous "
> + "deltacolumn or a mergepolicy other than
> snapshot", feed.getName()));
> }
> } else {
> throw new ValidationException(String.format("Feed %s is using
> unsupported "
> + "extraction mechanism %s", feed.getName(),
> extract.getType().value()));
> }
> }
> {code}
> It must have a clause to also check [incremental, append] .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FALCON-2234) Falcon feed fails for RDBMS import if extract type is incremental
[
https://issues.apache.org/jira/browse/FALCON-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804728#comment-15804728
]
Ajay Yadava commented on FALCON-2234:
-
Hello Prashant,
Thank you for reporting this issue. If I recall correctly, the original plan
was to add support for [incremental, append] as another patch which never came
through and hence has landed the documentation and validation in an
inconsistent state.
I believe the documentation needs to be fixed. Would you like to provide a
patch?
> Falcon feed fails for RDBMS import if extract type is incremental
> -
>
> Key: FALCON-2234
> URL: https://issues.apache.org/jira/browse/FALCON-2234
> Project: Falcon
> Issue Type: Bug
> Components: falcon-ui
>Affects Versions: 0.10
>Reporter: Prashant
>Priority: Critical
>
> Falcon feed is failing with
> org.apache.falcon.entity.parser.ValidationException
> when extract type is incremental.
> As per the Apache doc Falcon supports
> Import policy
> The valid combinations are:
> [full,snapshot] - data is extracted in full and dumped into the feed instance
> location.
> [incremental, append] - data is extracted incrementally using the key
> specified in the deltacolumn
> As per the below code , it appears we only check [full,snapshot] and
> [incremental, append] goes to exception
> https://github.com/apache/falcon/blob/0.10/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
> Snippet
> {code}
> /**
> * Validate extraction and merge type combination. Currently supported
> combo:
> *
> * ExtractionType = FULL and MergeType = SNAPSHOT.
> * ExtractionType = INCREMENTAL and MergeType = APPEND.
> *
> * @param feed Feed entity
> * @param cluster Cluster referenced in the Feed definition
> * @throws FalconException
> */
> private void validateFeedExtractionType(Feed feed, Cluster cluster)
> throws FalconException {
> Extract extract = cluster.getImport().getSource().getExtract();
> if (ExtractMethod.FULL == extract.getType()) {
> if ((MergeType.SNAPSHOT != extract.getMergepolicy())
> || (extract.getDeltacolumn() != null)) {
> throw new ValidationException(String.format("Feed %s is using
> FULL "
> + "extract method but specifies either a superfluous "
> + "deltacolumn or a mergepolicy other than
> snapshot", feed.getName()));
> }
> } else {
> throw new ValidationException(String.format("Feed %s is using
> unsupported "
> + "extraction mechanism %s", feed.getName(),
> extract.getType().value()));
> }
> }
> {code}
> It must have a clause to also check [incremental, append] .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FALCON-2236) Feed lifecycle retries for timeout
[ https://issues.apache.org/jira/browse/FALCON-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804710#comment-15804710 ] Ajay Yadava commented on FALCON-2236: - Sorry, I didn't understand what this JIRA is for. Can you please add a description of the issue? > Feed lifecycle retries for timeout > -- > > Key: FALCON-2236 > URL: https://issues.apache.org/jira/browse/FALCON-2236 > Project: Falcon > Issue Type: Improvement > Components: feed-lifecycle >Reporter: Pracheer Agarwal >Assignee: Pracheer Agarwal > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2014) Issue with idempotency in distributed mode
[ https://issues.apache.org/jira/browse/FALCON-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2014: Assignee: sandeep samudrala (was: Praveen Adlakha) > Issue with idempotency in distributed mode > -- > > Key: FALCON-2014 > URL: https://issues.apache.org/jira/browse/FALCON-2014 > Project: Falcon > Issue Type: Bug >Reporter: Pallavi Rao >Assignee: sandeep samudrala > > Lets say, a user submits an entity, entity1, against cluster1 and cluster2. > Later, he submits the same entity, but with a different definition, against > cluster3. Prism routes the request to cluster3 Falcon server. Since that > server does not have entity1, it updates its config store (with a new > definition). > However, prism later fails the submit after a definition check (as it does > not match with prev definition). Now, prism, cluster1 and cluster2 have > definition1 and cluster3 has definition2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Apache Falcon 0.11 release - Scope
+1. Thanks for driving this Praveen. On Sun, Dec 25, 2016 at 11:26 PM Sandeep Samudrala <sandys...@gmail.com> wrote: > +1 > Looks Good. Thanks for taking up the release. > > On Mon, Dec 26, 2016 at 9:45 AM, Pallavi Rao <pallavi@inmobi.com> > wrote: > > > Praveen, > > +1. > > > > How about adding feed retention based on number of instances (using > > lifecycle framework) to the list? > > > > Thanks, > > Pallavi > > > > On Sun, Dec 25, 2016 at 10:57 AM, Venkat Ranganathan < > > vranganat...@hortonworks.com> wrote: > > > > > +1 > > > > > > Thanks for taking up the release > > > > > > Venkat > > > > > > From: Jean-Baptiste Onofré <j...@nanthrax.net> > > > Sent: Saturday, December 24, 2016 1:29 AM > > > To: dev@falcon.apache.org > > > Subject: Re: Apache Falcon 0.11 release - Scope > > > > > > Hi, > > > > > > it sounds good to me. > > > > > > Thanks, > > > Regards > > > JB > > > > > > On 12/24/2016 10:05 AM, praveen adlakha wrote: > > > > Hi All, > > > > > > > > Apache Falcon team aims to make frequent releases. In keeping up with > > > this > > > > commitment, I think the following features are ready to be rolled out > > for > > > > the 0.11 release. > > > > > > > > > > > >- Support of Non-Trusted extensions in Distributed Mode/Embedded > > Mode > > > [ > > > >FALCON-2182 <https://issues.apache.org/jira/browse/FALCON-2182>]. > > > >- Backlog Metrics in graphite[FALCON-2170 > > > ><https://issues.apache.org/jira/browse/FALCON-2170>]. > > > >- Falcon Shell support for entity and instance > commands[FALCON-1596 > > > ><https://issues.apache.org/jira/browse/FALCON-1596>]. > > > >- Support to disable PostProcessing and moving logs from Server[ > > > >FALCON-2039 <https://issues.apache.org/jira/browse/FALCON-2039>]. > > > > > > > > > > > > Please let me know if there are any more features and enhancements > that > > > you > > > > wish to include in the release.Once the scope is freeze will come up > > with > > > > timelines. > > > > > > > > > > -- > > > Jean-Baptiste Onofré > > > jbono...@apache.org > > > http://blog.nanthrax.net > > > Talend - http://www.talend.com > > > > > > > > > > -- > > _ > > The information contained in this communication is intended solely for > the > > use of the individual or entity to whom it is addressed and others > > authorized to receive it. It may contain confidential or legally > privileged > > information. If you are not the intended recipient you are hereby > notified > > that any disclosure, copying, distribution or taking any action in > reliance > > on the contents of this information is strictly prohibited and may be > > unlawful. If you have received this communication in error, please notify > > us immediately by responding to this email and then delete it from your > > system. The firm is neither liable for the proper and complete > transmission > > of the information contained in this communication nor for any delay in > its > > receipt. > > > -- Regards Ajay Yadava
Volunteer for managing next release
Hello everyone, Due to some change in my situation, I am finding it a bit difficult to make time to coordinate with the community and drive the next release. It will be very helpful if someone else can take up the role of release manager for the next release. We have detailed(and tested) documentation <https://cwiki.apache.org/confluence/display/FALCON/How+to+do+an+Apache+Falcon+Release> on all the steps required to do a release, so hopefully, it should be pretty easy even if you have never done an Apache Falcon release before. Cheers Ajay Yadava
[jira] [Created] (FALCON-2204) Change mode for falcon_merge_pr.py to executable
Ajay Yadava created FALCON-2204: --- Summary: Change mode for falcon_merge_pr.py to executable Key: FALCON-2204 URL: https://issues.apache.org/jira/browse/FALCON-2204 Project: Falcon Issue Type: Improvement Components: dev-tools Reporter: Ajay Yadava Assignee: Ajay Yadava The script to merge pull requests is not executable on checkout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FALCON-2044) Persist Process stats in db
[ https://issues.apache.org/jira/browse/FALCON-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava resolved FALCON-2044. - Resolution: Fixed Fix Version/s: trunk Issue resolved by pull request 196 [https://github.com/apache/falcon/pull/196] > Persist Process stats in db > --- > > Key: FALCON-2044 > URL: https://issues.apache.org/jira/browse/FALCON-2044 > Project: Falcon > Issue Type: Improvement >Reporter: Praveen Adlakha >Assignee: Praveen Adlakha > Fix For: trunk > > > Hi All, > We should persist stats like process name,colo,nominal time ,delay time and > processing time in db so that we can do trend analysis over the performance > of the process. > I am thinking to persist it in the falcon DB if anyone has better approach > regarding this please provide there input. > Thanks > Praveen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-1406) Effective time in Entity updates.
[ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690980#comment-15690980 ] Ajay Yadava commented on FALCON-1406: - Hi [~sandeep.samudrala], In the interest of saving time, I and Srikanth had a detailed offline discussion on why versioning is required. I hope it will be much easier for you to sync up with him and understand the scenarios in detail. It will be nice to capture all those scenarios in the design doc e.g. 1. How does it impact other entities? Why this feature does or doesn't make sense for them? 2. How will the following systems / features work after this change: - graph DB, relational databases maintaining instance states, SLA (feed and process both), backlog, alerting, monitoring, instance dependency, triage, instance listing, entity lineage, recipes, late re-run, and retries. Just to clarify, I am not implying any changes to these systems, just want the design doc to capture your thoughts and discussion on these aspects, and we can make progress from there. More work coming your way :P On a serious note, really appreciate your effort and thanks for taking it up. It is a huge ask and has been a heroic effort to bring it up to this level. > Effective time in Entity updates. > - > > Key: FALCON-1406 > URL: https://issues.apache.org/jira/browse/FALCON-1406 > Project: Falcon > Issue Type: New Feature >Reporter: sandeep samudrala >Assignee: sandeep samudrala > Attachments: FALCON-1406-initial.patch, > effective_time_in_entity_updates.pdf > > > Effective time with entity updates needs to be provided even with past time > too. There was effective time capability provided in the past which gives the > functionality to set an effective time for an entity with only current or > future time(now + delay), which could not solve all the issues. > Following are few scenarios which would require effective time to be > available with time back in past. > a) New code being deployed for an incompatible input data set which would > leave instances with old code and new data. > b) Bad code being pushed for which, the entity should be able to go back in > time to replay(rerun) with new code. > c) Orchestration level changes(good/bad) would need functionality to go back > in time to start with. > For reference: Linking all the Jiras that have been worked upon around > effective time . > https://issues.apache.org/jira/browse/FALCON-374 > https://issues.apache.org/jira/browse/FALCON-297 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-1406) Effective time in Entity updates.
[
https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690243#comment-15690243
]
Ajay Yadava commented on FALCON-1406:
-
{quote} a lot of features developed in falcon took the immutability of *entity
definition* for past instances as given {quote}
Rerunning old instances doesn't affect that. New entities create new instances
and don't affect "old instances", though they might overwrite the result of
other entities.
{quote}This is a much cleaner way of retaining history than the current scheme
{quote}
I understand and agree with the motivation, but IMHO, this approach pollutes
history. The fact that the overlapping instances ran and had a status and are
selectively nuked with this change, is seemingly clean but is actually creating
more problems than solving. A better approach in retaining history will be to
create entity versioning.
The workaround approach that you have suggested now, is the one which doesn't
leave "mess" (defunct and similarly named entities) so you don't need a
cleanup. However, there are a lot of other cases where there will be such
"mess" and a tool can definitely be built to highlight such entities, older
than a given time range, and hence can be deleted. It will also be useful if
someone takes the approach of not deleting and recreating the entity but to
update the entity and reprocess old instances with a backfill job. Both
approaches have their own pros and cons and none is ideal.
> Effective time in Entity updates.
> -
>
> Key: FALCON-1406
> URL: https://issues.apache.org/jira/browse/FALCON-1406
> Project: Falcon
> Issue Type: New Feature
>Reporter: sandeep samudrala
>Assignee: sandeep samudrala
> Attachments: FALCON-1406-initial.patch,
> effective_time_in_entity_updates.pdf
>
>
> Effective time with entity updates needs to be provided even with past time
> too. There was effective time capability provided in the past which gives the
> functionality to set an effective time for an entity with only current or
> future time(now + delay), which could not solve all the issues.
> Following are few scenarios which would require effective time to be
> available with time back in past.
> a) New code being deployed for an incompatible input data set which would
> leave instances with old code and new data.
> b) Bad code being pushed for which, the entity should be able to go back in
> time to replay(rerun) with new code.
> c) Orchestration level changes(good/bad) would need functionality to go back
> in time to start with.
> For reference: Linking all the Jiras that have been worked upon around
> effective time .
> https://issues.apache.org/jira/browse/FALCON-374
> https://issues.apache.org/jira/browse/FALCON-297
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FALCON-1406) Effective time in Entity updates.
[ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687308#comment-15687308 ] Ajay Yadava commented on FALCON-1406: - Hi [~sriksun] I know that you want to see this feature go live for a long time, but with all sincerest intentions I believe that Falcon is better off without this change :) I think we all can agree that this change doesn't provide any new remarkable capabilities (capabilities that would have helped someone evaluating falcon for their use case and found it short) but just a way of doing things without creating temporary entities and probably more conveniently. Temporary entities are not created just for the reasons solved by this feature, they will be created even after this change e.g. during colo migrations, avoiding restrictions of moving start date to an earlier time and many more. It doesn't impact the system in any way, however, if you prefer to keep system "clean", then providing a utiliity to "cleanse" the system of such entities is much better and something which was proposed by the operations people who are the most affected by this "mess". The fact that an entity definition can change for a past instance is a rewrite of history and in my view is a violation of fundamental "assumption". Apart from just being a general bad practice IMHO, a lot of features developed in falcon took the immutability of entity definition for past instances as given. I suspect the lack of details in design doc has kept you from the true nature/impact of the change. I hope I am able to surface some of them once [~sandeep.samudrala] updates the design doc with other qustions that I have asked. Now, we can start fixing those issues on a case by case basis or we can just reevaluate the trade off of this change in favor of a command to clean up(addresses much wider use cases) or versioning of the entities. > Effective time in Entity updates. > - > > Key: FALCON-1406 > URL: https://issues.apache.org/jira/browse/FALCON-1406 > Project: Falcon > Issue Type: New Feature >Reporter: sandeep samudrala >Assignee: sandeep samudrala > Attachments: FALCON-1406-initial.patch, > effective_time_in_entity_updates.pdf > > > Effective time with entity updates needs to be provided even with past time > too. There was effective time capability provided in the past which gives the > functionality to set an effective time for an entity with only current or > future time(now + delay), which could not solve all the issues. > Following are few scenarios which would require effective time to be > available with time back in past. > a) New code being deployed for an incompatible input data set which would > leave instances with old code and new data. > b) Bad code being pushed for which, the entity should be able to go back in > time to replay(rerun) with new code. > c) Orchestration level changes(good/bad) would need functionality to go back > in time to start with. > For reference: Linking all the Jiras that have been worked upon around > effective time . > https://issues.apache.org/jira/browse/FALCON-374 > https://issues.apache.org/jira/browse/FALCON-297 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-1406) Effective time in Entity updates.
[ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684355#comment-15684355 ] Ajay Yadava commented on FALCON-1406: - [~sandeep.samudrala] I am still not in favor of this JIRA. My concern is not that you have made more changes than necessary, but rather it changes a fundamental assumption about the way falcon works and hence impacts a lot of current features and lot of features in future. This feature makes the code hard to reason for a lot of features. > Effective time in Entity updates. > - > > Key: FALCON-1406 > URL: https://issues.apache.org/jira/browse/FALCON-1406 > Project: Falcon > Issue Type: New Feature >Reporter: sandeep samudrala >Assignee: sandeep samudrala > Attachments: FALCON-1406-initial.patch, > effective_time_in_entity_updates.pdf > > > Effective time with entity updates needs to be provided even with past time > too. There was effective time capability provided in the past which gives the > functionality to set an effective time for an entity with only current or > future time(now + delay), which could not solve all the issues. > Following are few scenarios which would require effective time to be > available with time back in past. > a) New code being deployed for an incompatible input data set which would > leave instances with old code and new data. > b) Bad code being pushed for which, the entity should be able to go back in > time to replay(rerun) with new code. > c) Orchestration level changes(good/bad) would need functionality to go back > in time to start with. > For reference: Linking all the Jiras that have been worked upon around > effective time . > https://issues.apache.org/jira/browse/FALCON-374 > https://issues.apache.org/jira/browse/FALCON-297 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-1406) Effective time in Entity updates.
[ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684359#comment-15684359 ] Ajay Yadava commented on FALCON-1406: - Can you please also document what all steps are taken as part of an update with a backward effective time? Something on the lines of what other features have you checked that they won't be impacted and what steps have you taken for those which are impacted? e.g. what all changes are carried out in graph / relational db It will be nice to update the design doc with all these details as well. > Effective time in Entity updates. > - > > Key: FALCON-1406 > URL: https://issues.apache.org/jira/browse/FALCON-1406 > Project: Falcon > Issue Type: New Feature >Reporter: sandeep samudrala >Assignee: sandeep samudrala > Attachments: FALCON-1406-initial.patch, > effective_time_in_entity_updates.pdf > > > Effective time with entity updates needs to be provided even with past time > too. There was effective time capability provided in the past which gives the > functionality to set an effective time for an entity with only current or > future time(now + delay), which could not solve all the issues. > Following are few scenarios which would require effective time to be > available with time back in past. > a) New code being deployed for an incompatible input data set which would > leave instances with old code and new data. > b) Bad code being pushed for which, the entity should be able to go back in > time to replay(rerun) with new code. > c) Orchestration level changes(good/bad) would need functionality to go back > in time to start with. > For reference: Linking all the Jiras that have been worked upon around > effective time . > https://issues.apache.org/jira/browse/FALCON-374 > https://issues.apache.org/jira/browse/FALCON-297 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-1406) Effective time in Entity updates.
[ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684356#comment-15684356 ] Ajay Yadava commented on FALCON-1406: - Also, it will be nice to put changes of sub-tasks like FALCON-2169 in the main task, as without them feature doesn't work in all scenarios. This will ensure that the next release doesn't get blocked if one sub-task goes and other doesn't. > Effective time in Entity updates. > - > > Key: FALCON-1406 > URL: https://issues.apache.org/jira/browse/FALCON-1406 > Project: Falcon > Issue Type: New Feature >Reporter: sandeep samudrala >Assignee: sandeep samudrala > Attachments: FALCON-1406-initial.patch, > effective_time_in_entity_updates.pdf > > > Effective time with entity updates needs to be provided even with past time > too. There was effective time capability provided in the past which gives the > functionality to set an effective time for an entity with only current or > future time(now + delay), which could not solve all the issues. > Following are few scenarios which would require effective time to be > available with time back in past. > a) New code being deployed for an incompatible input data set which would > leave instances with old code and new data. > b) Bad code being pushed for which, the entity should be able to go back in > time to replay(rerun) with new code. > c) Orchestration level changes(good/bad) would need functionality to go back > in time to start with. > For reference: Linking all the Jiras that have been worked upon around > effective time . > https://issues.apache.org/jira/browse/FALCON-374 > https://issues.apache.org/jira/browse/FALCON-297 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 51424: Effective Time in Entity Update
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51424/#review154848 --- I haven't looked at the patch in detail, but I have noticed several things from earlier reviews. 1. It affects an unusually large number of functionalities. 2. It has taken a heroic effort over a considerable time to bring it up to this stage. 3. It adds lot of semantic burden on existing code base. 4. It will put an extra burden on the new features as they will have to support these scenarios as well. Given the ROI of this feature, all these are alarming signs. In my several years of supporting a large deployment of Falcon, I have never exeperienced a strong need for this as there are alternatives. IMO, it is a good to have but not a must have feature and given the ROI, I will recommend dropping it. If you have different views on why this feature is a must have, I am happy to hear them. - Ajay Yadava On Oct. 19, 2016, 6:43 p.m., sandeep samudrala wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51424/ > --- > > (Updated Oct. 19, 2016, 6:43 p.m.) > > > Review request for Falcon and Pallavi Rao. > > > Repository: falcon-git > > > Description > --- > > Effective Time in Entity Update > > > Diffs > - > > cli/src/main/java/org/apache/falcon/cli/FalconCLI.java 0dd11f6 > cli/src/main/java/org/apache/falcon/cli/FalconEntityCLI.java a8aea52 > client/src/main/java/org/apache/falcon/client/AbstractFalconClient.java > 5d6eff5 > client/src/main/java/org/apache/falcon/client/FalconCLIConstants.java > 04f1599 > client/src/main/java/org/apache/falcon/client/FalconClient.java 8f77fad > common/src/main/java/org/apache/falcon/entity/ClusterHelper.java f89def3 > common/src/main/java/org/apache/falcon/entity/EntityDictionaryUtil.java > PRE-CREATION > common/src/main/java/org/apache/falcon/entity/EntityLibEntry.java > PRE-CREATION > common/src/main/java/org/apache/falcon/entity/EntityUtil.java 8fe316c > common/src/main/java/org/apache/falcon/entity/ProcessHelper.java e563d18 > > common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java > 38fa3ae > common/src/main/java/org/apache/falcon/update/UpdateHelper.java 266319f > > common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java > 16a1753 > common/src/test/java/org/apache/falcon/entity/EntityDictionaryUtilTest.java > PRE-CREATION > common/src/test/java/org/apache/falcon/update/UpdateHelperTest.java 826686f > docs/src/site/twiki/falconcli/Touch.twiki afbd848 > docs/src/site/twiki/falconcli/UpdateEntity.twiki 146a60f > docs/src/site/twiki/restapi/EntityUpdate.twiki cbf33db > oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 5f93cc2 > oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java > c758411 > > oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java > 9f9579c > > oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java > f93a599 > > oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java > a1a7c12 > > oozie/src/main/java/org/apache/falcon/oozie/process/ProcessBundleBuilder.java > 6661dd5 > > oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java > 91f4757 > > oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java > 20eeffd > > oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java > 394600c > > oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java > 05b513e > oozie/src/test/resources/config/process/dumb-hive-process.xml c504074 > oozie/src/test/resources/config/process/hive-process-FSInputFeed.xml > d871377 > oozie/src/test/resources/config/process/hive-process-FSOutputFeed.xml > 23d96c3 > oozie/src/test/resources/config/process/hive-process.xml 4dac8e9 > oozie/src/test/resources/config/process/pig-process-0.1.xml 8d20cee > prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java > aefd699 > > prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java > 3bdeb99 > > prism/src/main/java/org/apache/falcon/resource/extensions/ExtensionManager.java > 92b5531 > > prism/src/main/java/org/apach
Re: [ANNOUNCE] New Committer Praveen Adlakha
Congratulations Praveen! Well earned. Cheers Ajay Yadava > On 11/3/16, 7:51 AM, "Srikanth Sundarrajan" <srik...@hotmail.com> > wrote: > > >The Project Management Committee (PMC) for Apache Falcon has asked > >Praveen Adlakha to become a committer and we are pleased to announce > that > >he has accepted. > > > >Regards > >Srikanth Sundarrajan (on behalf of Apache Falcon PMC) > > > > >
Request to reschedule dev sync up
Hi, I have moved to a different time zone and the current scheduled time is very late in night for me. Can we please reschedule the sync up so that I can also join? IST 4 pm to 7am is what suits me. In the meantime can we move the sync up to next week if no one has urgent issues to be discussed? Regards Ajay Yadava
[jira] [Commented] (FALCON-2141) Bump up pom to 1.0
[ https://issues.apache.org/jira/browse/FALCON-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464888#comment-15464888 ] Ajay Yadava commented on FALCON-2141: - [~pallavi.rao], [~sandeep.samudrala] I hope I answered your questions, let me know if you see any issues. > Bump up pom to 1.0 > -- > > Key: FALCON-2141 > URL: https://issues.apache.org/jira/browse/FALCON-2141 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > As discussed in an earlier dev-sync up, we will do development for 1.0 in > master. This JIRA is to update the poms in the project to 1.0 instead of 0.11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[DISCUSS] Commit guidelines
Hello everyone, Over time, I have observed a few patterns and I feel that there are certain areas where we can collectively improve on. Following is the list of guidelines that I have put together. - Don't commit patches without docs. - Don't commit patches without test-cases - Ensure release notes are updated in JIRA wherever applicable. *[new]* Please don't use / indulge separate JIRAs for docs / tests. All code changes should go in along with tests and docs. - Don't commit unless all review comments are addressed / answered. Please ensure all review comments, including ones from other contributors (not just committers') are addressed. As much as possible, avoid creating separate JIRAs for review comments. - Encourage reuse of pull requests Sometimes contributors unintentionally create a new pull request instead of updating the old one where the review comments were provided. Contributors should strive to reuse the old prs in order to preserve the history of reviews. Committers should try to check that prs are reused and old feedback is addressed. - Avoid putting meaningless squashed commit messages in the detailed commit message. The pr-merge tool has an option to list the squashed commit messages as part of the detailed commit message. Most pull requests have meaningless intermediate commit messages, please don't include them in the detailed commit message. However, if commit messages add valuable context, please include them. May be we should change the default option to no in the tool. - Give others a chance to review. IMHO, it is polite to wait at least 24 Hrs after a +1 before committing a change. If yours is first +1 and you intend to commit it, please consider putting a comment expressing the same. This gives others who may have some opinions on the pull request, but couldn't get to it before you, a chance to review. - Update fixVersions and update thoughtfully. Again using pr-merge tool here helps ensuring that fixVersion is updated, though figuring out correct fixVersion requires some explanation. Simple rule of thumb is - change belongs to the earliest release in which it got committed e.g. if a release branch has been cut and a commit is applied to both master and release branch, fixVersion should be release version (and not trunk). - Commit changes to all applicable branches A common mistake is to commit the patch only to the branch and not to master. Although, this has been reduced a lot by the pr tool, but it is still happening and something to keep in mind when you are not using the pr-merge tool. Just to clarify, this list is not meant to serve as list of rules but as a set of guidelines. Like all good things, these are not expected to be followed blindly. Part of being a committer is to develop the art of knowing when to make an exception. A simple comment explaining your rationale goes a long way ;) As usual, suggestions / modifications / addenda are welcome. Regards Ajay Yadava
[jira] [Resolved] (FALCON-2144) Close stale pull requests - part 2
[ https://issues.apache.org/jira/browse/FALCON-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava resolved FALCON-2144. - Resolution: Fixed Done! All shiny and clean now :) @Committers Please take a look at the [board|https://github.com/apache/falcon/pulls] and let's review and merge the valid ones now. > Close stale pull requests - part 2 > -- > > Key: FALCON-2144 > URL: https://issues.apache.org/jira/browse/FALCON-2144 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > Sorry I missed one in the earlier patch which I thought the author closed and > found another one. > https://github.com/apache/falcon/pull/33 > https://github.com/apache/falcon/pull/258 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2144) Close stale pull requests - part 2
[ https://issues.apache.org/jira/browse/FALCON-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2144: Summary: Close stale pull requests - part 2 (was: Close stale pull requests - part2) > Close stale pull requests - part 2 > -- > > Key: FALCON-2144 > URL: https://issues.apache.org/jira/browse/FALCON-2144 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > > Sorry I missed one in the earlier patch which I thought the author closed and > found another one. > https://github.com/apache/falcon/pull/33 > https://github.com/apache/falcon/pull/258 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FALCON-2144) Close stale pull requests - part2
Ajay Yadava created FALCON-2144: --- Summary: Close stale pull requests - part2 Key: FALCON-2144 URL: https://issues.apache.org/jira/browse/FALCON-2144 Project: Falcon Issue Type: Task Reporter: Ajay Yadava Assignee: Ajay Yadava Sorry I missed one in the earlier patch which I thought the author closed and found another one. https://github.com/apache/falcon/pull/33 https://github.com/apache/falcon/pull/258 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FALCON-2142) Close stale pull requests
[ https://issues.apache.org/jira/browse/FALCON-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava resolved FALCON-2142. - Resolution: Fixed Done! > Close stale pull requests > - > > Key: FALCON-2142 > URL: https://issues.apache.org/jira/browse/FALCON-2142 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > Several pull requests have changes which are already merged / not required or > need some work but have been lying unresponsive for quite some time, despite > reminders. The list of these pull requests is as below: > https://github.com/apache/falcon/pull/3 > https://github.com/apache/falcon/pull/4 > https://github.com/apache/falcon/pull/5 > https://github.com/apache/falcon/pull/7 > https://github.com/apache/falcon/pull/10 > https://github.com/apache/falcon/pull/26 > https://github.com/apache/falcon/pull/32 > https://github.com/apache/falcon/pull/33 > https://github.com/apache/falcon/pull/34 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2142) Close stale pull requests
[ https://issues.apache.org/jira/browse/FALCON-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460354#comment-15460354 ] Ajay Yadava commented on FALCON-2142: - Thank you everyone, who closed their pull requests. I will go ahead and close the remaining now. > Close stale pull requests > - > > Key: FALCON-2142 > URL: https://issues.apache.org/jira/browse/FALCON-2142 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > Several pull requests have changes which are already merged / not required or > need some work but have been lying unresponsive for quite some time, despite > reminders. The list of these pull requests is as below: > https://github.com/apache/falcon/pull/3 > https://github.com/apache/falcon/pull/4 > https://github.com/apache/falcon/pull/5 > https://github.com/apache/falcon/pull/7 > https://github.com/apache/falcon/pull/10 > https://github.com/apache/falcon/pull/26 > https://github.com/apache/falcon/pull/32 > https://github.com/apache/falcon/pull/33 > https://github.com/apache/falcon/pull/34 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2141) Bump up pom to 1.0
[ https://issues.apache.org/jira/browse/FALCON-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460346#comment-15460346 ] Ajay Yadava commented on FALCON-2141: - [~pallavi.rao] There are already several 1.0 features like Falcon Shell, Post processing changes etc. which are merged to trunk, so no use in cutting a branch from HEAD now. If we need to roll out critical fixes then we can cut the branch from 0.10 and apply patches to that. [~sandeep.samudrala] All our releases take longer than expected.. haha! The idea of interim release(if required) is to roll out critical bug fixes and not a feature release :) We should cut a branch once the community decides on an interim release. You can always cut it from any commit in past, so it shouldn't be a blocker in any case :) > Bump up pom to 1.0 > -- > > Key: FALCON-2141 > URL: https://issues.apache.org/jira/browse/FALCON-2141 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > As discussed in an earlier dev-sync up, we will do development for 1.0 in > master. This JIRA is to update the poms in the project to 1.0 instead of 0.11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Apache Falcon 1.0
Hello Srikanth, Excellent point. I think everyone is in favor of API stability and versioning irrespective of 1.0 or not, however I suspect it may mean different things for different people(e.g. backward incompatible?). I think it will be useful to have that discussion again in open to bring everyone on same page. Would you like to start a discuss thread with summary of your thoughts? Regards Ajay Yadava On Fri, Sep 2, 2016 at 9:55 AM Srikanth Sundarrajan <srik...@hotmail.com> wrote: > Am +1 for 1.0 release and also feel that newer and more attractive > features can follow. I dont know if there was any outcome on the approach > to API versioning discussion that we had earlier this year, we should > re-initiate that discussion and close if there is general consensus for 1.0. > RegardsSrikanth Sundarrajan > > > From: ajayyad...@apache.org > > Date: Thu, 1 Sep 2016 14:55:53 + > > Subject: [DISCUSS] Apache Falcon 1.0 > > To: dev@falcon.apache.org > > > > Hello everyone, > > > > Now that 0.10 has been successfully released, thanks to heroic efforts of > > Balu, I think it is time to start discussion on our long pending 1.0 > > release :) I specifically want to discuss the following items. > > > > *Scope / Change Threshold for calling it 1.0* > > > > We need to decide on the bare minimum changes to call it 1.0 I know at > > least 2 views on this in the community > > > >1. First view is that 1.0 is all about API stability, Once we have > >stabilized the existing API, we can call it 1.0. There are already > some > >changes committed to trunk like a new shell, moving post processing to > >server side, REST API Revamp and some more lifecycle changes which > will > >definitely come in 1.0 This should be enough to do a meaningful 1.0 > release. > >2. Another view is that we need to have lot of marketable features and > >new capabilities to call it 1.0 so that we can publicize it enough > and ask > >audience to reconsider falcon even if they found Apache Falcon short > for > >their use cases. > > > > Please chime in with your thoughts :) > > > > > > *Call for Volunteer* > > > > As discussed in last developer sync up, Pragya communicated to me that > she > > has very high work load at her office and will not be able to take out > time > > for release activities. If there are any committers who haven't yet done > a > > release and want to volunteer for 1.0, then please reply to this thread, > > else I will assume the role of Release Manager for 1.0. Many release > > activities require karma available only to committers, hence only > > committers can volunteer. > > > > > > Regards > > Ajay Yadava >
[jira] [Updated] (FALCON-2141) Bump up pom to 1.0
[ https://issues.apache.org/jira/browse/FALCON-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2141: Summary: Bump up pom to 1.0 (was: Update pom to 1.0 in master) > Bump up pom to 1.0 > -- > > Key: FALCON-2141 > URL: https://issues.apache.org/jira/browse/FALCON-2141 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > As discussed in an earlier dev-sync up, we will do development for 1.0 in > master. This JIRA is to update the poms in the project to 1.0 instead of 0.11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2141) Update pom to 1.0 in master
[ https://issues.apache.org/jira/browse/FALCON-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2141: Fix Version/s: 1.0 > Update pom to 1.0 in master > --- > > Key: FALCON-2141 > URL: https://issues.apache.org/jira/browse/FALCON-2141 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > As discussed in an earlier dev-sync up, we will do development for 1.0 in > master. This JIRA is to update the poms in the project to 1.0 instead of 0.11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2142) Close stale pull requests
[ https://issues.apache.org/jira/browse/FALCON-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455736#comment-15455736 ] Ajay Yadava commented on FALCON-2142: - This is a harmless change as the authors can reopen the pull request any time they want. However, I will wait till tomorrow to give everyone a chance to review the list. If no one objects then I will go ahead and close the pull requests in the list. > Close stale pull requests > - > > Key: FALCON-2142 > URL: https://issues.apache.org/jira/browse/FALCON-2142 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > Several pull requests have changes which are already merged / not required or > need some work but have been lying unresponsive for quite some time, despite > reminders. The list of these pull requests is as below: > https://github.com/apache/falcon/pull/3 > https://github.com/apache/falcon/pull/4 > https://github.com/apache/falcon/pull/5 > https://github.com/apache/falcon/pull/7 > https://github.com/apache/falcon/pull/10 > https://github.com/apache/falcon/pull/26 > https://github.com/apache/falcon/pull/32 > https://github.com/apache/falcon/pull/33 > https://github.com/apache/falcon/pull/34 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2142) Close stale pull requests
[ https://issues.apache.org/jira/browse/FALCON-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2142: Fix Version/s: 1.0 > Close stale pull requests > - > > Key: FALCON-2142 > URL: https://issues.apache.org/jira/browse/FALCON-2142 > Project: Falcon > Issue Type: Task > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 1.0 > > > Several pull requests have changes which are already merged / not required or > need some work but have been lying unresponsive for quite some time, despite > reminders. The list of these pull requests is as below: > https://github.com/apache/falcon/pull/3 > https://github.com/apache/falcon/pull/4 > https://github.com/apache/falcon/pull/5 > https://github.com/apache/falcon/pull/7 > https://github.com/apache/falcon/pull/10 > https://github.com/apache/falcon/pull/26 > https://github.com/apache/falcon/pull/32 > https://github.com/apache/falcon/pull/33 > https://github.com/apache/falcon/pull/34 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FALCON-2142) Close stale pull requests
Ajay Yadava created FALCON-2142: --- Summary: Close stale pull requests Key: FALCON-2142 URL: https://issues.apache.org/jira/browse/FALCON-2142 Project: Falcon Issue Type: Task Reporter: Ajay Yadava Assignee: Ajay Yadava Several pull requests have changes which are already merged / not required or need some work but have been lying unresponsive for quite some time, despite reminders. The list of these pull requests is as below: https://github.com/apache/falcon/pull/3 https://github.com/apache/falcon/pull/4 https://github.com/apache/falcon/pull/5 https://github.com/apache/falcon/pull/7 https://github.com/apache/falcon/pull/10 https://github.com/apache/falcon/pull/26 https://github.com/apache/falcon/pull/32 https://github.com/apache/falcon/pull/33 https://github.com/apache/falcon/pull/34 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FALCON-2141) Update pom to 1.0 in master
Ajay Yadava created FALCON-2141: --- Summary: Update pom to 1.0 in master Key: FALCON-2141 URL: https://issues.apache.org/jira/browse/FALCON-2141 Project: Falcon Issue Type: Task Reporter: Ajay Yadava Assignee: Ajay Yadava As discussed in an earlier dev-sync up, we will do development for 1.0 in master. This JIRA is to update the poms in the project to 1.0 instead of 0.11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[DISCUSS] Apache Falcon 1.0
Hello everyone, Now that 0.10 has been successfully released, thanks to heroic efforts of Balu, I think it is time to start discussion on our long pending 1.0 release :) I specifically want to discuss the following items. *Scope / Change Threshold for calling it 1.0* We need to decide on the bare minimum changes to call it 1.0 I know at least 2 views on this in the community 1. First view is that 1.0 is all about API stability, Once we have stabilized the existing API, we can call it 1.0. There are already some changes committed to trunk like a new shell, moving post processing to server side, REST API Revamp and some more lifecycle changes which will definitely come in 1.0 This should be enough to do a meaningful 1.0 release. 2. Another view is that we need to have lot of marketable features and new capabilities to call it 1.0 so that we can publicize it enough and ask audience to reconsider falcon even if they found Apache Falcon short for their use cases. Please chime in with your thoughts :) *Call for Volunteer* As discussed in last developer sync up, Pragya communicated to me that she has very high work load at her office and will not be able to take out time for release activities. If there are any committers who haven't yet done a release and want to volunteer for 1.0, then please reply to this thread, else I will assume the role of Release Manager for 1.0. Many release activities require karma available only to committers, hence only committers can volunteer. Regards Ajay Yadava
[jira] [Commented] (FALCON-2080) Exceptions in the log
[
https://issues.apache.org/jira/browse/FALCON-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454379#comment-15454379
]
Ajay Yadava commented on FALCON-2080:
-
Changed fix version to 1.0 for some testing purposes.
> Exceptions in the log
> -
>
> Key: FALCON-2080
> URL: https://issues.apache.org/jira/browse/FALCON-2080
> Project: Falcon
> Issue Type: Bug
>Reporter: Praveen Adlakha
>Assignee: Praveen Adlakha
> Fix For: 1.0
>
>
> Hi All,
> We are getting the following exceptions in the logs :
> {code}
> ouldn't find status for Entity:ProcessInstanceRerunTest-agregator-10,
> cluster:local, entityTypePROCESS, nominalTimeWed Jul 13 15:35:00 IST 2016
> (FeedSLA:473)
> org.apache.falcon.FalconException: java.lang.IllegalStateException: No user
> logged into the system : java.lang.IllegalStateException: No user logged into
> the system
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.findBundles(OozieWorkflowEngine.java:362)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.findBundles(OozieWorkflowEngine.java:372)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.getCoordActions(OozieWorkflowEngine.java:1104)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.doJobAction(OozieWorkflowEngine.java:666)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.doJobAction(OozieWorkflowEngine.java:737)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.getStatus(OozieWorkflowEngine.java:598)
> at
> org.apache.falcon.service.EntitySLAMonitoringService.checkEntityInstanceAvailability(EntitySLAMonitoringService.java:451)
> at
> org.apache.falcon.service.EntitySLAMonitoringService.checkPendingInstanceAvailability(EntitySLAMonitoringService.java:430)
> at
> org.apache.falcon.service.EntitySLAMonitoringService.access$200(EntitySLAMonitoringService.java:72)
> at
> org.apache.falcon.service.EntitySLAMonitoringService$Monitor.run(EntitySLAMonitoringService.java:349)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FALCON-2080) Exceptions in the log
[
https://issues.apache.org/jira/browse/FALCON-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Yadava updated FALCON-2080:
Fix Version/s: (was: trunk)
1.0
> Exceptions in the log
> -
>
> Key: FALCON-2080
> URL: https://issues.apache.org/jira/browse/FALCON-2080
> Project: Falcon
> Issue Type: Bug
>Reporter: Praveen Adlakha
>Assignee: Praveen Adlakha
> Fix For: 1.0
>
>
> Hi All,
> We are getting the following exceptions in the logs :
> {code}
> ouldn't find status for Entity:ProcessInstanceRerunTest-agregator-10,
> cluster:local, entityTypePROCESS, nominalTimeWed Jul 13 15:35:00 IST 2016
> (FeedSLA:473)
> org.apache.falcon.FalconException: java.lang.IllegalStateException: No user
> logged into the system : java.lang.IllegalStateException: No user logged into
> the system
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.findBundles(OozieWorkflowEngine.java:362)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.findBundles(OozieWorkflowEngine.java:372)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.getCoordActions(OozieWorkflowEngine.java:1104)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.doJobAction(OozieWorkflowEngine.java:666)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.doJobAction(OozieWorkflowEngine.java:737)
> at
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.getStatus(OozieWorkflowEngine.java:598)
> at
> org.apache.falcon.service.EntitySLAMonitoringService.checkEntityInstanceAvailability(EntitySLAMonitoringService.java:451)
> at
> org.apache.falcon.service.EntitySLAMonitoringService.checkPendingInstanceAvailability(EntitySLAMonitoringService.java:430)
> at
> org.apache.falcon.service.EntitySLAMonitoringService.access$200(EntitySLAMonitoringService.java:72)
> at
> org.apache.falcon.service.EntitySLAMonitoringService$Monitor.run(EntitySLAMonitoringService.java:349)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FALCON-1440) Better json for triage api
[
https://issues.apache.org/jira/browse/FALCON-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Yadava updated FALCON-1440:
Fix Version/s: 1.0
> Better json for triage api
> --
>
> Key: FALCON-1440
> URL: https://issues.apache.org/jira/browse/FALCON-1440
> Project: Falcon
> Issue Type: Sub-task
>Reporter: Raghav Kumar Gautam
> Assignee: Ajay Yadava
> Fix For: 1.0
>
>
> Firing rest request for triage returns a response which looks like:
> {code}
> "vertices": [
> "name: A78e9f5a1-5b8eab89, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [AVAILABLE]"
> ]
> {code}
> The cli output can also be formatted better:
> {code}
> digraph g{
> "name: A78e9f5a1-83173830, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [MISSING]"
> }
> {code}
> Here is the full request/response that was made:
> instance/triage/feed/A78e9f5a1-5b8eab89?start=2010-01-02T00%3A40Z=2015-01-05T01%3A00Z=hrt_qa
> {code}
> {
> "triageGraphs": [
> {
> "vertices": [
> "name: A78e9f5a1-5b8eab89, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [AVAILABLE]"
> ]
> }
> ],
> "requestId": "default/1712257401@qtp-1961945640-622 -
> 4b19ad04-2433-41c8-ad11-541fc0509e0d\n",
> "message": "default/Success\n",
> "status": "SUCCEEDED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Democratizing Release Notes
Hello everyone, Continuing on our effort to automating change log and release notes, I am happy to share with you that we now have the ability to automatically generate release notes through Apache Yetus. *How does it work?* We have introduced a new field in JIRA - Release Notes. Every contributor needs to add release notes for their important / backward-incompatible changes / new features. Later, RMs can generate release notes using Apache Yetus. Many projects like Hadoop etc. have already been doing it this way. We have also been using Apache Yetus to generate the release notes *Why?* Generating release notes had been one tedious task which the RMs(Release Managers) had to do manually at the end of the release by going through all JIRAs and writing notes for them. Often, RM may not even be the best person to write release notes for the feature. *Note to Contributors* Please add release notes for any important / backward-incompatible changes / new features that you add. *Note to Committers* Please don't commit PRs for important / backward-incompatible changes / new features without release notes. I will update the "how to release" wiki with the steps for RMs, In the meantime, you can consult the documentation for Apache Yetus here <https://yetus.apache.org/documentation/0.3.0/> Regards Ajay Yadava
[jira] [Commented] (FALCON-1440) Better json for triage api
[
https://issues.apache.org/jira/browse/FALCON-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448221#comment-15448221
]
Ajay Yadava commented on FALCON-1440:
-
I wanted to ensure backward compatibility so I was waiting for starting to work
on next version of REST API to do this.
> Better json for triage api
> --
>
> Key: FALCON-1440
> URL: https://issues.apache.org/jira/browse/FALCON-1440
> Project: Falcon
> Issue Type: Sub-task
>Reporter: Raghav Kumar Gautam
> Assignee: Ajay Yadava
>
> Firing rest request for triage returns a response which looks like:
> {code}
> "vertices": [
> "name: A78e9f5a1-5b8eab89, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [AVAILABLE]"
> ]
> {code}
> The cli output can also be formatted better:
> {code}
> digraph g{
> "name: A78e9f5a1-83173830, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [MISSING]"
> }
> {code}
> Here is the full request/response that was made:
> instance/triage/feed/A78e9f5a1-5b8eab89?start=2010-01-02T00%3A40Z=2015-01-05T01%3A00Z=hrt_qa
> {code}
> {
> "triageGraphs": [
> {
> "vertices": [
> "name: A78e9f5a1-5b8eab89, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [AVAILABLE]"
> ]
> }
> ],
> "requestId": "default/1712257401@qtp-1961945640-622 -
> 4b19ad04-2433-41c8-ad11-541fc0509e0d\n",
> "message": "default/Success\n",
> "status": "SUCCEEDED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FALCON-1440) Better json for triage api
[
https://issues.apache.org/jira/browse/FALCON-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Yadava updated FALCON-1440:
Issue Type: Sub-task (was: Bug)
Parent: FALCON-1871
> Better json for triage api
> --
>
> Key: FALCON-1440
> URL: https://issues.apache.org/jira/browse/FALCON-1440
> Project: Falcon
> Issue Type: Sub-task
>Reporter: Raghav Kumar Gautam
> Assignee: Ajay Yadava
>
> Firing rest request for triage returns a response which looks like:
> {code}
> "vertices": [
> "name: A78e9f5a1-5b8eab89, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [AVAILABLE]"
> ]
> {code}
> The cli output can also be formatted better:
> {code}
> digraph g{
> "name: A78e9f5a1-83173830, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [MISSING]"
> }
> {code}
> Here is the full request/response that was made:
> instance/triage/feed/A78e9f5a1-5b8eab89?start=2010-01-02T00%3A40Z=2015-01-05T01%3A00Z=hrt_qa
> {code}
> {
> "triageGraphs": [
> {
> "vertices": [
> "name: A78e9f5a1-5b8eab89, type: FEED, cluster: A78e9f5a1-9238cc6e,
> instanceTime: 2010-01-02T00:40Z, tags: [AVAILABLE]"
> ]
> }
> ],
> "requestId": "default/1712257401@qtp-1961945640-622 -
> 4b19ad04-2433-41c8-ad11-541fc0509e0d\n",
> "message": "default/Success\n",
> "status": "SUCCEEDED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FALCON-475) Add Version to the REST APIs
[ https://issues.apache.org/jira/browse/FALCON-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-475: --- Issue Type: Sub-task (was: Improvement) Parent: FALCON-1871 > Add Version to the REST APIs > > > Key: FALCON-475 > URL: https://issues.apache.org/jira/browse/FALCON-475 > Project: Falcon > Issue Type: Sub-task > Components: webapp >Affects Versions: 0.6 >Reporter: Venkatesh Seetharam > Labels: rest_api > > Versioning is necessary since REST APIs are planned to be driven from state > persisted in a DB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FALCON-475) Add Version to the REST APIs
[ https://issues.apache.org/jira/browse/FALCON-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava reassigned FALCON-475: -- Assignee: Ajay Yadava > Add Version to the REST APIs > > > Key: FALCON-475 > URL: https://issues.apache.org/jira/browse/FALCON-475 > Project: Falcon > Issue Type: Sub-task > Components: webapp >Affects Versions: 0.6 >Reporter: Venkatesh Seetharam >Assignee: Ajay Yadava > Labels: rest_api > > Versioning is necessary since REST APIs are planned to be driven from state > persisted in a DB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (FALCON-1406) Effective time in Entity updates.
[ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-1406: Comment: was deleted (was: [~sandeep.samudrala] Please update the old review request instead of creating a new one, otherwise the old context and review comments get lost after the update. For now, I have given my comments, on the new review board request. Some correctness and functional issues which I noticed and I feel are worth capturing here for everyone's consideration. 1. This feature will break entity sla features as it contains some instances for monitoring sla and they might be affected with entity update(and sla update along with it) 2. Touch command is also a way to do an update, however it is not updated to accept effectiveTime parameter. 3. This feature will also not work in cases where input feed instances may have been deleted by retention and on passing effectiveTime from past those instances will not succeed. 4. Instance search feature will also break by these changes as we are not taking care of existing graph db instances. 5. After several effective updates, it will be hard to keep track of which workflow/code worked with which input/jars. ) > Effective time in Entity updates. > - > > Key: FALCON-1406 > URL: https://issues.apache.org/jira/browse/FALCON-1406 > Project: Falcon > Issue Type: New Feature >Reporter: sandeep samudrala >Assignee: sandeep samudrala > Attachments: FALCON-1406-initial.patch, > effective_time_in_entity_updates.pdf > > > Effective time with entity updates needs to be provided even with past time > too. There was effective time capability provided in the past which gives the > functionality to set an effective time for an entity with only current or > future time(now + delay), which could not solve all the issues. > Following are few scenarios which would require effective time to be > available with time back in past. > a) New code being deployed for an incompatible input data set which would > leave instances with old code and new data. > b) Bad code being pushed for which, the entity should be able to go back in > time to replay(rerun) with new code. > c) Orchestration level changes(good/bad) would need functionality to go back > in time to start with. > For reference: Linking all the Jiras that have been worked upon around > effective time . > https://issues.apache.org/jira/browse/FALCON-374 > https://issues.apache.org/jira/browse/FALCON-297 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-1406) Effective time in Entity updates.
[ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15443958#comment-15443958 ] Ajay Yadava commented on FALCON-1406: - [~sandeep.samudrala] Please update the old review request instead of creating a new one, otherwise the old context and review comments get lost after the update. For now, I have given my comments, on the new review board request. Some correctness and functional issues which I noticed and I feel are worth capturing here for everyone's consideration. 1. This feature will break entity sla features as it contains some instances for monitoring sla and they might be affected with entity update(and sla update along with it) 2. Touch command is also a way to do an update, however it is not updated to accept effectiveTime parameter. 3. This feature will also not work in cases where input feed instances may have been deleted by retention and on passing effectiveTime from past those instances will not succeed. 4. Instance search feature will also break by these changes as we are not taking care of existing graph db instances. 5. After several effective updates, it will be hard to keep track of which workflow/code worked with which input/jars. > Effective time in Entity updates. > - > > Key: FALCON-1406 > URL: https://issues.apache.org/jira/browse/FALCON-1406 > Project: Falcon > Issue Type: New Feature >Reporter: sandeep samudrala >Assignee: sandeep samudrala > Attachments: FALCON-1406-initial.patch, > effective_time_in_entity_updates.pdf > > > Effective time with entity updates needs to be provided even with past time > too. There was effective time capability provided in the past which gives the > functionality to set an effective time for an entity with only current or > future time(now + delay), which could not solve all the issues. > Following are few scenarios which would require effective time to be > available with time back in past. > a) New code being deployed for an incompatible input data set which would > leave instances with old code and new data. > b) Bad code being pushed for which, the entity should be able to go back in > time to replay(rerun) with new code. > c) Orchestration level changes(good/bad) would need functionality to go back > in time to start with. > For reference: Linking all the Jiras that have been worked upon around > effective time . > https://issues.apache.org/jira/browse/FALCON-374 > https://issues.apache.org/jira/browse/FALCON-297 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-1478) Enable feed lifecycle through native scheduler
[ https://issues.apache.org/jira/browse/FALCON-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-1478: Assignee: (was: Ajay Yadava) > Enable feed lifecycle through native scheduler > -- > > Key: FALCON-1478 > URL: https://issues.apache.org/jira/browse/FALCON-1478 > Project: Falcon > Issue Type: New Feature > Reporter: Ajay Yadava > > Once the native scheduler patch is committed, lifecycle needs to be > integrated through the native scheduler as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-1493) Email notifications in case of sla miss
[ https://issues.apache.org/jira/browse/FALCON-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-1493: Assignee: (was: Ajay Yadava) > Email notifications in case of sla miss > --- > > Key: FALCON-1493 > URL: https://issues.apache.org/jira/browse/FALCON-1493 > Project: Falcon > Issue Type: Sub-task > Reporter: Ajay Yadava > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FALCON-2119) Minor Licensing Issue
Ajay Yadava created FALCON-2119: --- Summary: Minor Licensing Issue Key: FALCON-2119 URL: https://issues.apache.org/jira/browse/FALCON-2119 Project: Falcon Issue Type: Improvement Reporter: Ajay Yadava Assignee: Ajay Yadava falcon-sources-0.10/falcon-ui/app/css/styles/angular-notify.less is actually not an original work and is exactly same as MIT Licensed (but not mentioned in LICENSE.txt) except for few commented lines. https://github.com/cgross/angular-notify/blob/master/angular-notify.css -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release candidate RC2 of Apache Falcon 0.10
Hi Balu, Sorry that I did not finish all the checks in RC0 itself or even now. I am preparing to relocate to a different continent and unable to give it the time it deserves to do a complete check. I would be happy to convert my vote to +1, if someone convinces me that the following are not an issue. 1. Bootstrap license I didn't notice it earlier and it seems that unlike earlier bootstrap(licensed only under MIT), library we have used has dual license (MIT and Apache Software License 2.0) and can be used under ASL 2.0 and hence probably need not mention in the LICENSE.txt. However, bootstrap includes *normalize.css* which is licensed under MIT only(see line 9 in bootstrap.min.css) which will still need to be mentioned in the LICENSE. 2. Entypo license *2. Entypo license is covered under line 242 in LICENSE.txt* INAL, but I believe Entypo is mentioned under LICENSE.txt but since it is CC BY-SA we need to update the NOTICE along with the License. 3. ** *All angular js related licenses are covered under line 229* ** This may really be a non-issue and you may be right here. My reason for bringing it up is because I suspect that these are not part of plain angular.min.js download and are separate helper packages from angular, which if correct, might require them to be mentioned separately in LICENSE, even though they have same origin and license. 4. Other files *./falcon-ui/app/js/lib/bootstrap.notify.js [MIT only] - no mention in LICENSE* not part of bootstrap distribution. Quoting from the source below /* * Project: Bootstrap Notify = v3.0.2 * Description: Turns standard Bootstrap alerts into "Growl-like" notifications. * Author: Mouse0270 aka Robert McIntosh * License: MIT License * Website: https://github.com/mouse0270/bootstrap-growl */ *./falcon-ui/app/js/lib/checklist-model.js * Following is the header: /** * Checklist-model * AngularJS directive for list of checkboxes */ It should have ASL header if it's original work contributed directly to Apache Falcon. Absence of Google copyright suggests it is not part of angular js either. *./falcon-ui/app/js/lib/popover.js * No ASL header, no other hints on source of origin either. FWIW, technically -1 is not a veto for releases and this vote may pass as per the guidelines [1] so you may not need to put out another RC. [1] http://www.apache.org/foundation/voting.html#ReleaseVotes Regards Ajay Yadava On Thu, Aug 4, 2016 at 2:42 AM Balu Vellanki Bala <bvella...@hortonworks.com> wrote: > Hi Ajay > > 1. There is docs/license/bootstrap-LICENSE.txt for bootstrap. Angular > bootstrap license is listed in LICENSE.txt at line 232, > "This product bundles angular-ui-bootstrap 0.11.0 which is available under > a > MIT license. For details, see > docs/license/angular-ui-bootstrap-LICENSE.txt" > > > All angular js related licenses are covered under line 229 > "This product bundles angularJS 1.3.5 which is available under a > MIT license. For details, see docs/license/angularJS-LICENSE.txt" > This product bundles entypo icons which is available under a > CC BY-SA license and Font is available under SIL license. > For details, see docs/license/entypo-icons-LICENSE.txt and > docs/license/entypo-font-LICENSE.txt" > > > If you agree, please recast your vote. > > Balu Vellanki > > On 8/3/16, 1:13 PM, "Ajay Yadava" <ajayyad...@apache.org> wrote: > > >-1 (binding) > > > >because: > >1. Following libraries are being bundled as part of the source but don't > >have any mention in the license > > > > - bootstrap > > - Following add on libraries for angular js are not distributed as part > > of angular.min.js and hence also deserve separate mention AFAIK [ > > angular-mocks.js, angular-messages.min.js, angular-cookies.min.js, > > angular-animate.min.js ] > > - checklist-model.js > > > >2. entypo icons are being used, which are licensed under CC-BY SA which I > >believe requires a mention in NOTICE also. > > > >I haven't yet looked at everything, will let you know if I find more > >issues. Thank you for driving this release Balu! > > > >Regards > >Ajay Yadava > > > >On Wed, Aug 3, 2016 at 7:19 PM Peeyush Bishnoi > ><bpeey...@yahoo.co.in.invalid> > >wrote: > > > >> Verified the release candidate > >> - Checksums (MD5, SHA) - MATCH > >> - Signature - GOOD > >> - Build - SUCCESSFUL (default & distributed) > >> +1 > >> > >> On Wednesday, 3 August 2016 12:53 PM, Jean-Baptiste Onofré < > >> j...@nanthrax.net> wrote: > >> > >> > >> +1 (binding) > >> > >> Build successfully. Legal files look OK. > >> > >> Reg
Re: [VOTE] Release candidate RC2 of Apache Falcon 0.10
-1 (binding) because: 1. Following libraries are being bundled as part of the source but don't have any mention in the license - bootstrap - Following add on libraries for angular js are not distributed as part of angular.min.js and hence also deserve separate mention AFAIK [ angular-mocks.js, angular-messages.min.js, angular-cookies.min.js, angular-animate.min.js ] - checklist-model.js 2. entypo icons are being used, which are licensed under CC-BY SA which I believe requires a mention in NOTICE also. I haven't yet looked at everything, will let you know if I find more issues. Thank you for driving this release Balu! Regards Ajay Yadava On Wed, Aug 3, 2016 at 7:19 PM Peeyush Bishnoi <bpeey...@yahoo.co.in.invalid> wrote: > Verified the release candidate > - Checksums (MD5, SHA) - MATCH > - Signature - GOOD > - Build - SUCCESSFUL (default & distributed) > +1 > > On Wednesday, 3 August 2016 12:53 PM, Jean-Baptiste Onofré < > j...@nanthrax.net> wrote: > > > +1 (binding) > > Build successfully. Legal files look OK. > > Regards > JB > > On 08/03/2016 09:22 AM, Pallavi Rao wrote: > > +1 > > > > Signature - OK > > Checksums - OK > > Build - OK > > Licenses - OK > > > > Thanks a bunch, Balu, for driving the release! > > > > > > On Wed, Aug 3, 2016 at 9:08 AM, Venkat Ranganathan < > > vranganat...@hortonworks.com> wrote: > > > >> +1 > >> > >> Built and ran tests successfully > >> Verified signatures and checksums > >> Validated licenses > >> > >> > >> One thing we should do going forward is to add some attention calling > >> sequence – like a line full of ‘*’ to point to the fact that CHANGES.txt > >> does not have changes from Release 0.10 onwards. > >> > >> Thanks Balu for driving the release and working through the issues > >> > >> Thanks > >> > >> Venkat > >> > >> > >> On 8/2/16, 4:32 PM, "Balu Vellanki Bala" <bvella...@hortonworks.com> > >> wrote: > >> > >>Hello Everyone, > >> > >>This is the call for vote for the following Release Candidate RC2 to > be > >>released as official Apache Falcon 0.10 release. This RC2 addressed > all > >>feedback provided for RC0, plus fixed bugs FALCON-2104 and > FALCON-2107 > >>found in RC0/RC1. The source tar ball, signature and checksum files > >> can be > >>downloaded from > >> > >> > https://dist.apache.org/repos/dist/dev/falcon/apache-falcon-0.10-sources/ > >> > >>You can find the signing key here : > >>http://pgp.mit.edu/pks/lookup?op=vindex=0xF743ABF42E887C76 > >> > >>You can find the KEYS file here: > >>https://dist.apache.org/repos/dist/release/falcon/KEYS > >> > >> > >>To verify signature: > >>gpg --verify apache-falcon-0.10-sources.tar.gz.asc > >>apache-falcon-0.10-sources.tar.gz > >> > >>Git tag commit-id for the release : release-0.10-rc2 > >> > >> > https://git-wip-us.apache.org/repos/asf?p=falcon.git;a=commit;h=dc4129598d9 > >>afed9ca9e4210d7c04f80e458ff3f > >> > >>We have closed around several JIRAs as part of this release, the > >> details > >>of which can be found here: > >> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12334758 > >>leName=Html=12314429 > >> > >> > >>Please find the release notes with brief explanation on key changes > at > >> > >> > https://issues.apache.org/jira/secure/attachment/12821716/FalconReleaseNote > >>s-0.10.pdf > >>Note: The hyper link ³migration instructions² under section-5 is set > to > >> > >> > https://svn.apache.org/repos/asf/falcon/site/0.10/MigrationInstructions.htm > >>l, this link will only be available post release, once the Falcon > >>documentation site is updated. At this moment, it will not work. > >> > >> > >> > >>This vote will remain open for at least 72 hours. Please vote on > >> releasing > >>this RC2 > >> > >>[ ] +1 approve > >>[ ] +0 no opinion > >>[ ] -1 disapprove (and reason why) > >> > >>+1 from my side > >> > >>Thank you > >>Balu Vellanki > >> > >> > >> > >> > >> > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com > > >
[jira] [Resolved] (FALCON-1596) Spring shell based CLI for falcon
[ https://issues.apache.org/jira/browse/FALCON-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava resolved FALCON-1596. - Resolution: Fixed Issue resolved by pull request 249 [https://github.com/apache/falcon/pull/249] > Spring shell based CLI for falcon > - > > Key: FALCON-1596 > URL: https://issues.apache.org/jira/browse/FALCON-1596 > Project: Falcon > Issue Type: New Feature > Components: shell >Reporter: Rajat Khandelwal >Assignee: Praveen Adlakha > Fix For: trunk > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2105) Documentation for Falcon Spring Shell
[ https://issues.apache.org/jira/browse/FALCON-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2105: Description: Add user documentation for the newly introduced shell. (was: Hi All, I am pretty much done with coding and testing of the spring shell this jira is for tracking the documentation for the same. Thanks Pravene) > Documentation for Falcon Spring Shell > - > > Key: FALCON-2105 > URL: https://issues.apache.org/jira/browse/FALCON-2105 > Project: Falcon > Issue Type: Improvement > Components: shell >Reporter: Praveen Adlakha >Assignee: Praveen Adlakha > > Add user documentation for the newly introduced shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2105) Documentation for Falcon Spring Shell
[ https://issues.apache.org/jira/browse/FALCON-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2105: Component/s: shell > Documentation for Falcon Spring Shell > - > > Key: FALCON-2105 > URL: https://issues.apache.org/jira/browse/FALCON-2105 > Project: Falcon > Issue Type: Improvement > Components: shell >Reporter: Praveen Adlakha >Assignee: Praveen Adlakha > > Hi All, > I am pretty much done with coding and testing of the spring shell this jira > is for tracking the documentation for the same. > Thanks > Pravene -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2105) Documentation for Falcon Spring Shell
[ https://issues.apache.org/jira/browse/FALCON-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403815#comment-15403815 ] Ajay Yadava commented on FALCON-2105: - [~Praveen], Can you please link it with the JIRA for the code? Also, (I know I am being greedy, hahaha) but will you be able to also add some screenshots and a short video? > Documentation for Falcon Spring Shell > - > > Key: FALCON-2105 > URL: https://issues.apache.org/jira/browse/FALCON-2105 > Project: Falcon > Issue Type: Improvement > Components: shell >Reporter: Praveen Adlakha >Assignee: Praveen Adlakha > > Hi All, > I am pretty much done with coding and testing of the spring shell this jira > is for tracking the documentation for the same. > Thanks > Pravene -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2059) BacklogMetricEmitter Service for Falcon
[ https://issues.apache.org/jira/browse/FALCON-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2059: Assignee: pavan kumar kolamuri (was: Ajay Yadava) > BacklogMetricEmitter Service for Falcon > > > Key: FALCON-2059 > URL: https://issues.apache.org/jira/browse/FALCON-2059 > Project: Falcon > Issue Type: Task >Reporter: pavan kumar kolamuri >Assignee: pavan kumar kolamuri > Fix For: trunk > > > Backlog Metric Emitter service evaluates backlog for processes and emits > backlog metrics to graphite -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2059) BacklogMetricEmitter Service for Falcon
[ https://issues.apache.org/jira/browse/FALCON-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400644#comment-15400644 ] Ajay Yadava commented on FALCON-2059: - Sorry, keyboard shortcut got accidentally activated. > BacklogMetricEmitter Service for Falcon > > > Key: FALCON-2059 > URL: https://issues.apache.org/jira/browse/FALCON-2059 > Project: Falcon > Issue Type: Task >Reporter: pavan kumar kolamuri > Assignee: Ajay Yadava > Fix For: trunk > > > Backlog Metric Emitter service evaluates backlog for processes and emits > backlog metrics to graphite -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FALCON-2059) BacklogMetricEmitter Service for Falcon
[ https://issues.apache.org/jira/browse/FALCON-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava reassigned FALCON-2059: --- Assignee: Ajay Yadava (was: pavan kumar kolamuri) > BacklogMetricEmitter Service for Falcon > > > Key: FALCON-2059 > URL: https://issues.apache.org/jira/browse/FALCON-2059 > Project: Falcon > Issue Type: Task >Reporter: pavan kumar kolamuri > Assignee: Ajay Yadava > Fix For: trunk > > > Backlog Metric Emitter service evaluates backlog for processes and emits > backlog metrics to graphite -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release of Apache Falcon 0.10 RC0
notes with brief explanation on key > >>changes > >> >>at > >> >> JIRA https://issues.apache.org/jira/browse/FALCON-2098 > >> >> > >> >>This vote will remain open for at least 72 hours. Please vote on > >> >> releasing this RC. > >> >><https://issues.apache.org/jira/browse/FALCON-2098> > >> >> > >> >>[ ] +1 approve > >> >>[ ] +0 no opinion > >> >>[ ] -1 disapprove (and reason why) > >> >> > >> >>+1 from my side > >> >> > >> >>Thank you > >> >>Balu Vellanki > >> >> > >> >> > >> >> > >> >> > >> > > >> >-- > >> >_ > >> >The information contained in this communication is intended solely for > >> >the > >> >use of the individual or entity to whom it is addressed and others > >> >authorized to receive it. It may contain confidential or legally > >> >privileged > >> >information. If you are not the intended recipient you are hereby > >> >notified > >> >that any disclosure, copying, distribution or taking any action in > >> >reliance > >> >on the contents of this information is strictly prohibited and may be > >> >unlawful. If you have received this communication in error, please > >>notify > >> >us immediately by responding to this email and then delete it from your > >> >system. The firm is neither liable for the proper and complete > >> >transmission > >> >of the information contained in this communication nor for any delay in > >> >its > >> >receipt. > >> > > >> > > >> > >> > > > >-- > >_ > >The information contained in this communication is intended solely for > >the > >use of the individual or entity to whom it is addressed and others > >authorized to receive it. It may contain confidential or legally > >privileged > >information. If you are not the intended recipient you are hereby > >notified > >that any disclosure, copying, distribution or taking any action in > >reliance > >on the contents of this information is strictly prohibited and may be > >unlawful. If you have received this communication in error, please notify > >us immediately by responding to this email and then delete it from your > >system. The firm is neither liable for the proper and complete > >transmission > >of the information contained in this communication nor for any delay in > >its > >receipt. > > -- Regards Ajay Yadava
Re: [VOTE] Release of Apache Falcon 0.10 RC0
Balu, Venkat, I haven't finished my checks but it seems I have hit a blocker. I verified following: - Verified the signature Was able to receive the key and verify the signature but *[1]* - Verified md5 and sha512 - Checked release Notes *[2]* - License and Notice *NOTICE* Copyright notice is 2015 and should be extended to 2016. *Licensing* There is a dependency on JTS which has LGPL license *[3]* AFAIK LGPL is incompatible and shouldn't be bundled. [1] Balu's key is not signed by anyone else so not in my WOT. [2] FALCON-1858 is a backward incompatible change because it has changed the default settings and will be surprising for existing users. It should be mentioned under incompatible changes in release notes. *[FALCON-1858] - Support HBase as a storage backend for Falcon Titan graphDB. Users will be able to choose between HBase (default) and the current Berkeley DB storage* [3] http://www.vividsolutions.com/jts/JTSHome.htm
[jira] [Updated] (FALCON-2063) Add change log for 0.10
[ https://issues.apache.org/jira/browse/FALCON-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2063: Fix Version/s: 0.10 > Add change log for 0.10 > --- > > Key: FALCON-2063 > URL: https://issues.apache.org/jira/browse/FALCON-2063 > Project: Falcon > Issue Type: New Feature > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 0.10 > > > As per the discussion on the mailing list we had decided to deprecate manual > additions to CHANGES.txt and generate it automatically. > I propose to use Apache YETUS for generating CHANGES and release notes and it > has following benefits. > 1. Generating changes.txt from jira is much more flexible than from commit > messages. As I noted earlier also, commits can't be modified, so if a > developer forgets to attach metadata / attaches wrong metadata then > generating release notes will be incorrect. > 2. Yetus provides nice markdown formatted change log with links to each > issue. They are easy to read and you can go to the jira by clicking through > the github. > 3. It can also automate the generation of release notes, if we add them to > corresponding JIRA. > 4. We don't need to write / maintain and go through all the existing/future > learning that YETUS went through. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FALCON-2063) Add change log for 0.10
[ https://issues.apache.org/jira/browse/FALCON-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava resolved FALCON-2063. - Resolution: Fixed > Add change log for 0.10 > --- > > Key: FALCON-2063 > URL: https://issues.apache.org/jira/browse/FALCON-2063 > Project: Falcon > Issue Type: New Feature > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Fix For: 0.10 > > > As per the discussion on the mailing list we had decided to deprecate manual > additions to CHANGES.txt and generate it automatically. > I propose to use Apache YETUS for generating CHANGES and release notes and it > has following benefits. > 1. Generating changes.txt from jira is much more flexible than from commit > messages. As I noted earlier also, commits can't be modified, so if a > developer forgets to attach metadata / attaches wrong metadata then > generating release notes will be incorrect. > 2. Yetus provides nice markdown formatted change log with links to each > issue. They are easy to read and you can go to the jira by clicking through > the github. > 3. It can also automate the generation of release notes, if we add them to > corresponding JIRA. > 4. We don't need to write / maintain and go through all the existing/future > learning that YETUS went through. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Jenkins build failures for Apache Falcon
I am of the opinion that such tests are more suitable for regression. These tests take lot of time and make running tests locally or debugging them a huge pain. We built falcon-unit for such cases but adoption is very low and even in that we have seen large time outs. Tests which have time outs of 10 seconds are not suitable for pre commit checks. It also creates a bad entry barrier for new users / contributors to just download the source and run all tests locally. We should probably schedule a nightly / weekly build of regression and that will be more thorough and helpful in such scenarios. On Fri, Jul 22, 2016 at 10:30 AM Ying Zheng <yzh...@hortonworks.com> wrote: > +1. > > On 7/21/16, 9:34 PM, "Venkat Ranganathan" <vranganat...@hortonworks.com> > wrote: > > >Thanks Srikanth for giving access to Balu.Last time I looked at the > >jobs, there seems to be a dependency on the specific host where it was > >running. Trying to move it another host was not helpful. May be we > >make it run on a pool of machines than one as it is currently done > > > >Venkat > > > >On 7/21/16, 6:58 PM, "Srikanth Sundarrajan" <srik...@hotmail.com> wrote: > > > >Hi Balu,Physical access to the machine may be harder. But it is > >possible to modify the jenkins job to put out any logs from the build > >apart from the binaries it produces. Hopefully that should allow us to > >debug this issue. I can either make changes you seek to the build or give > >you permissions to modify the job yourself. The later may be a better > >choice. > >RegardsSrikanth Sundarrajan > > > >> Subject: Jenkins build failures for Apache Falcon > >> From: bvella...@hortonworks.com > >> To: dev@falcon.apache.org > >> Date: Thu, 21 Jul 2016 23:30:08 + > >> > >> Hi Team, > >> > >> Recently, builds are failing for Apache Falcon (Email Ref : Build > >failed > >> in Jenkins: Apache-falcon #1064). The tests that fail require > >instances > >> scheduled in workflow engine to reach a certain state like RUNNING > >or > >> SUCCEEDED. But the instances seem to be stuck in state WAITING even > >after > >> delays of up to 10 seconds. > >> > >> The same tests do not fail in local environment. The machine running > >> Jenkins builds does not seem to have enough resources. I request > >access to > >> the machine for further debugging and adjusting resource allocation > >OR > >> will accept help from some one who has access to the machine. > >> > >> Please let me know, > >> Thank you > >> Balu Vellanki > >> > > > > > > -- Regards Ajay Yadava
[jira] [Updated] (FALCON-2063) Add change log for 0.10
[ https://issues.apache.org/jira/browse/FALCON-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2063: Summary: Add change log for 0.10 (was: Add change log for each release ) > Add change log for 0.10 > --- > > Key: FALCON-2063 > URL: https://issues.apache.org/jira/browse/FALCON-2063 > Project: Falcon > Issue Type: New Feature > Reporter: Ajay Yadava > Assignee: Ajay Yadava > > As per the discussion on the mailing list we had decided to deprecate manual > additions to CHANGES.txt and generate it automatically. > I propose to use Apache YETUS for generating CHANGES and release notes and it > has following benefits. > 1. Generating changes.txt from jira is much more flexible than from commit > messages. As I noted earlier also, commits can't be modified, so if a > developer forgets to attach metadata / attaches wrong metadata then > generating release notes will be incorrect. > 2. Yetus provides nice markdown formatted change log with links to each > issue. They are easy to read and you can go to the jira by clicking through > the github. > 3. It can also automate the generation of release notes, if we add them to > corresponding JIRA. > 4. We don't need to write / maintain and go through all the existing/future > learning that YETUS went through. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Tasks remaining for 0.10 release candidate
Hi All, When is the next fortnightly syncup? On Mon, Jul 18, 2016 at 12:56 PM Balu Vellanki Bala < bvella...@hortonworks.com> wrote: > Hi Pragya and team > > At this point all issues have been fixed on 0.10 and you I am hoping you > had time to verify that there are no more issues. I will create RC-0 > Monday pacific time. > > Thank you > Balu Vellanki > > On 7/14/16, 5:54 AM, "Pallavi Rao" <pallavi@inmobi.com> wrote: > > >Balu, et. al, > >The status of the blocker bugs raised by Pragya are as follows: > >FALCON-2070 - Resolved as Invalid > >FALCON-2076, FALCON-1749, FALCON-2067 - Merged to trunk and 0.10 > >FALCON-2060 - In progress. Should be done by morrow. > > > >Once FALCON-2060 is merged, Pragya can do the final verification and > >sign-off. > > > >Thanks, > >Pallavi > > > > > >On Wed, Jul 13, 2016 at 12:24 PM, Venkat Ranganathan < > >vranganat...@hortonworks.com> wrote: > > > >> Thanks Balu > >> > >> Sorry I missed the meeting. I was stuck in another work session and > >> could not get out of it quickly. > >> > >> It looks like we are converging. Hopefully we should be able to do a > >>RC > >> for voting this week! > >> > >> Thanks > >> > >> Venkat > >> > >> > >> On 7/12/16, 9:50 PM, "Balu Vellanki Bala" <bvella...@hortonworks.com> > >> wrote: > >> > >> Hi Team, > >> > >> We had bi-weekly Falcon collaboration meeting and discussed the 0.10 > >> release candidate. The following tasks came out of meeting, > >> > >> > >> 1. Pragya opened Jiras for issues that caused regression or feature > >> failure in 0.10. Pallavi and Balu to work on these issues within next > >>two > >> days. > >> 2. Balu will update 0.9 Jiras to be Apache Yetus compliant > >> 3. Regenerate changelog for Apache Falcon branches. > >> 4. Pragya to re-run the regression tests asap after tasks 1 and 3 are > >> done. > >> > >> We hope to have a release candidate by Friday pacific time if all goes > >> well, > >> Thank you > >> Balu Vellanki > >> > >> > >> > > > >-- > >_ > >The information contained in this communication is intended solely for > >the > >use of the individual or entity to whom it is addressed and others > >authorized to receive it. It may contain confidential or legally > >privileged > >information. If you are not the intended recipient you are hereby > >notified > >that any disclosure, copying, distribution or taking any action in > >reliance > >on the contents of this information is strictly prohibited and may be > >unlawful. If you have received this communication in error, please notify > >us immediately by responding to this email and then delete it from your > >system. The firm is neither liable for the proper and complete > >transmission > >of the information contained in this communication nor for any delay in > >its > >receipt. > > -- Regards Ajay Yadava
[jira] [Created] (FALCON-2064) Show more helpful error message in case of missing dependency
Ajay Yadava created FALCON-2064: --- Summary: Show more helpful error message in case of missing dependency Key: FALCON-2064 URL: https://issues.apache.org/jira/browse/FALCON-2064 Project: Falcon Issue Type: Improvement Reporter: Ajay Yadava Assignee: Ajay Yadava Although it is mentioned in the documentation, it will be helpful to users to provide error message if requirements like python-dateutil are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FALCON-2063) Add change log for each release
Ajay Yadava created FALCON-2063: --- Summary: Add change log for each release Key: FALCON-2063 URL: https://issues.apache.org/jira/browse/FALCON-2063 Project: Falcon Issue Type: New Feature Reporter: Ajay Yadava Assignee: Ajay Yadava As per the discussion on the mailing list we had decided to deprecate manual additions to CHANGES.txt and generate it automatically. I propose to use Apache YETUS for generating CHANGES and release notes and it has following benefits. 1. Generating changes.txt from jira is much more flexible than from commit messages. As I noted earlier also, commits can't be modified, so if a developer forgets to attach metadata / attaches wrong metadata then generating release notes will be incorrect. 2. Yetus provides nice markdown formatted change log with links to each issue. They are easy to read and you can go to the jira by clicking through the github. 3. It can also automate the generation of release notes, if we add them to corresponding JIRA. 4. We don't need to write / maintain and go through all the existing/future learning that YETUS went through. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2062) Fix Checkstyle issues in trunk
[ https://issues.apache.org/jira/browse/FALCON-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365928#comment-15365928 ] Ajay Yadava commented on FALCON-2062: - Sorry for the inconvenience, will raise a pull request shortly. > Fix Checkstyle issues in trunk > -- > > Key: FALCON-2062 > URL: https://issues.apache.org/jira/browse/FALCON-2062 > Project: Falcon > Issue Type: Bug > Reporter: Ajay Yadava > Assignee: Ajay Yadava > > While testing pull/14 I forgot to use test-patch profile and some checkstyle > issues creeped in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-2062) Fix Checkstyle issues in trunk
[ https://issues.apache.org/jira/browse/FALCON-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-2062: Description: While merging pull/14 I forgot to use test-patch profile during testing and some checkstyle issues creeped in. (was: While testing pull/14 I forgot to use test-patch profile and some checkstyle issues creeped in. ) > Fix Checkstyle issues in trunk > -- > > Key: FALCON-2062 > URL: https://issues.apache.org/jira/browse/FALCON-2062 > Project: Falcon > Issue Type: Bug > Reporter: Ajay Yadava > Assignee: Ajay Yadava > > While merging pull/14 I forgot to use test-patch profile during testing and > some checkstyle issues creeped in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FALCON-2062) Fix Checkstyle issues in trunk
Ajay Yadava created FALCON-2062: --- Summary: Fix Checkstyle issues in trunk Key: FALCON-2062 URL: https://issues.apache.org/jira/browse/FALCON-2062 Project: Falcon Issue Type: Bug Reporter: Ajay Yadava Assignee: Ajay Yadava While testing pull/14 I forgot to use test-patch profile and some checkstyle issues creeped in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-1691) Falcon 0.9 release
[ https://issues.apache.org/jira/browse/FALCON-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-1691: Fix Version/s: (was: 0.9) 0.10 > Falcon 0.9 release > -- > > Key: FALCON-1691 > URL: https://issues.apache.org/jira/browse/FALCON-1691 > Project: Falcon > Issue Type: Task >Reporter: Pallavi Rao >Assignee: Pallavi Rao > Fix For: 0.10 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-1801) Update CHANGES.txt in trunk to mark 0.9 as released
[ https://issues.apache.org/jira/browse/FALCON-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-1801: Summary: Update CHANGES.txt in trunk to mark 0.9 as released (was: Update CHANGES.txt in trunk from proposed release to actual release version) > Update CHANGES.txt in trunk to mark 0.9 as released > --- > > Key: FALCON-1801 > URL: https://issues.apache.org/jira/browse/FALCON-1801 > Project: Falcon > Issue Type: Sub-task >Reporter: Pallavi Rao >Assignee: Pallavi Rao > Fix For: trunk > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FALCON-2052) Process SLA monitoring
[ https://issues.apache.org/jira/browse/FALCON-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava resolved FALCON-2052. - Resolution: Fixed Fix Version/s: trunk Issue resolved by pull request 202 [https://github.com/apache/falcon/pull/202] > Process SLA monitoring > --- > > Key: FALCON-2052 > URL: https://issues.apache.org/jira/browse/FALCON-2052 > Project: Falcon > Issue Type: Improvement >Reporter: Praveen Adlakha >Assignee: Praveen Adlakha > Fix For: trunk > > > Hi All, > Like we have implemented Feed SLA monitoring we should have process SLA > monitoring also.The design on the approach will pretty much be same as that > of FeedSLA,will be making necessary changes in that code base too. > Thanks > Praveen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
[ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344233#comment-15344233 ] Ajay Yadava edited comment on FALCON-2030 at 6/22/16 12:34 PM: --- [~bvellanki], [~me.venkatr] I think I understood where the disconnect is. Allow me to explain. Each retention policy has it's own validation, if some retention policy needs feed to be in time partition pattern then we should do validation and throw error. However, this shouldn't be a blanket check and if the user is not using some feature which makes sense only for time based partition, this should be allowed. We are planning to have more such retention policies where feeds don't need to be in time partition style, in next release of Falcon and then users won't need to write custom retention lifecycles. Same holds for import/export case as well. If the feed without time partition doesn't make sense for them, import / export should have such checks but if someone is not using import/export those validations shouldn't kick in. Makes sense? was (Author: ajayyadava): [~bvellanki], [~me.venkatr] I think I understood where the disconnect is. Allow me to explain. Each retention policy has it's own validation, if some retention policy needs feed to be in time partition pattern then we should do validation and throw error. However, this shouldn't be a blanket check and if the user is not using some feature which makes sense only for time based partition, this should be allowed. We are planning to have more such retention policies where feeds don't need to be in time partition style, in next release of Falcon and then users won't need to write custom retention lifecycles. Same holds for import/export case as well. If it doesn't make sense for them, import / export should have such checks but if someone is not using import/export those validations shouldn't kick in. Makes sense? > Enforce time partition pattern in the data location path in feed definition > > > Key: FALCON-2030 > URL: https://issues.apache.org/jira/browse/FALCON-2030 > Project: Falcon > Issue Type: Improvement > Components: feed >Reporter: Venkatesan Ramachandran >Assignee: Venkatesan Ramachandran > > In feed definition, data location can be specified without time series > pattern like below: > > path="/tmp/falcon-regression/RetentionTest/testFolders/"/> > > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
[ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339391#comment-15339391 ] Ajay Yadava commented on FALCON-2030: - [~venkatnrangan] I am under impression that after FALCON-2023 there is no scenario where a user will be caught off-guard by lack of this validation. Is there another use case which is still not addressed? > Enforce time partition pattern in the data location path in feed definition > > > Key: FALCON-2030 > URL: https://issues.apache.org/jira/browse/FALCON-2030 > Project: Falcon > Issue Type: Improvement > Components: feed >Reporter: Venkatesan Ramachandran >Assignee: Venkatesan Ramachandran > > In feed definition, data location can be specified without time series > pattern like below: > > path="/tmp/falcon-regression/RetentionTest/testFolders/"/> > > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
[ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335368#comment-15335368 ] Ajay Yadava edited comment on FALCON-2030 at 6/17/16 6:32 AM: -- Retention deletes on the basis of creation time or last accessed time. Like retention the consumers, *if any* exist, will also have to consume only on the base of creation time/access time to avoid consuming files scheduled for deletion. Use case may also be as simple as a way to automatically delete temporary/unused files/directories from a location and no consumers as such. was (Author: ajayyadava): Retention deletes on the basis of creation time or last accessed time. Like retention the consumers, *if any* exist, will also have to consume only on the base of creation time/access time to avoid deletion. Use case may also be as simple as a way to automatically delete temporary/unused files/directories from a location. > Enforce time partition pattern in the data location path in feed definition > > > Key: FALCON-2030 > URL: https://issues.apache.org/jira/browse/FALCON-2030 > Project: Falcon > Issue Type: Improvement > Components: feed >Reporter: Venkatesan Ramachandran >Assignee: Venkatesan Ramachandran > > In feed definition, data location can be specified without time series > pattern like below: > > path="/tmp/falcon-regression/RetentionTest/testFolders/"/> > > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
[ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335368#comment-15335368 ] Ajay Yadava commented on FALCON-2030: - Retention deletes on the basis of creation time or last accessed time. Like retention the consumers, *if any* exist, will also have to consume only on the base of creation time/access time to avoid deletion. Use case may also be as simple as a way to automatically delete temporary/unused files/directories from a location. > Enforce time partition pattern in the data location path in feed definition > > > Key: FALCON-2030 > URL: https://issues.apache.org/jira/browse/FALCON-2030 > Project: Falcon > Issue Type: Improvement > Components: feed >Reporter: Venkatesan Ramachandran >Assignee: Venkatesan Ramachandran > > In feed definition, data location can be specified without time series > pattern like below: > > path="/tmp/falcon-regression/RetentionTest/testFolders/"/> > > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
[ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334259#comment-15334259 ] Ajay Yadava commented on FALCON-2030: - I understand the motivation to help users but I think the fix in FALCON-2023 is correct, sufficient and forward looking approach. We should fix any issues which are caused by relying on the feeds locations to conform a particular pattern. The date pattern should be to facilitate certain use cases but not to block any other use cases. Let me give another practical benefit of *not* enforcing the pattern. Falcon users can now write their own custom retention policy, so one can now write a custom policy which treats a folder as the feed location and deletes only the sub folders or files with creation time older than a certain date. In fact this was the use case for a certain user on the mailing list. > Enforce time partition pattern in the data location path in feed definition > > > Key: FALCON-2030 > URL: https://issues.apache.org/jira/browse/FALCON-2030 > Project: Falcon > Issue Type: Improvement > Components: feed >Reporter: Venkatesan Ramachandran >Assignee: Venkatesan Ramachandran > > In feed definition, data location can be specified without time series > pattern like below: > > path="/tmp/falcon-regression/RetentionTest/testFolders/"/> > > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
[ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334247#comment-15334247 ] Ajay Yadava commented on FALCON-2030: - Thanks [~me.venkatr]! Metadatas do change but it is not necessary to maintain their versions as it is not practically useful to maintain old versions. Only the latest and one source of truth is required and maintained. Also, as you said that users might use a version number, a unique id like epoch etc., or just use the english version like /mydata-june. How will we enforce the pattern in those cases? > Enforce time partition pattern in the data location path in feed definition > > > Key: FALCON-2030 > URL: https://issues.apache.org/jira/browse/FALCON-2030 > Project: Falcon > Issue Type: Improvement > Components: feed >Reporter: Venkatesan Ramachandran >Assignee: Venkatesan Ramachandran > > In feed definition, data location can be specified without time series > pattern like below: > > path="/tmp/falcon-regression/RetentionTest/testFolders/"/> > > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
[ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331570#comment-15331570 ] Ajay Yadava commented on FALCON-2030: - There is one use case - a metadata snapshot whose only one instance is maintained. The limiting factor is not Oozie as Oozie doesn't mandate it to have timeseries pattern. However, IIRC retention will have problem in such cases but use case requires that metadata should never be deleted so people use something like years(999). It is not uncommon to have such metadata use cases e.g. typically in ad-networks metadata like device metadata, ip metadata etc. are used like that. [~me.venkatr] Won't it be the case with incremental snapshots in import as well? I have been away for a while so not aware of the exact motivation to force this validation. Are there any issues because of this? > Enforce time partition pattern in the data location path in feed definition > > > Key: FALCON-2030 > URL: https://issues.apache.org/jira/browse/FALCON-2030 > Project: Falcon > Issue Type: Improvement > Components: feed >Reporter: Venkatesan Ramachandran >Assignee: Venkatesan Ramachandran > > In feed definition, data location can be specified without time series > pattern like below: > > path="/tmp/falcon-regression/RetentionTest/testFolders/"/> > > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FALCON-291) Not able to package Falcon
[
https://issues.apache.org/jira/browse/FALCON-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242649#comment-15242649
]
Ajay Yadava edited comment on FALCON-291 at 4/15/16 8:41 AM:
-
Hi jp,
{quote}
curl: (6) Couldn't resolve host 'archive.apache.org'
{quote}
This seems to indicate that there was some problem accessing the above url.
Basically build is trying to download
http://archive.apache.org/dist/oozie/4.1.0/oozie-4.1.0.tar.gz but it fails.
Can you try accessing this url in your browser and let us know if you are able
to open it?
was (Author: ajayyadava):
Hi jp,
{quote}
curl: (6) Couldn't resolve host 'archive.apache.org'
{quote}
This seems to indicate that there was some problem accessing the above url.
Basically build is trying to download
http://archive.apache.org/dist/oozie/4.1.0/oozie-4.1.0.tar.gz but it fails.
Can you try accessing this url and let us know if you are able to open it?
> Not able to package Falcon
> --
>
> Key: FALCON-291
> URL: https://issues.apache.org/jira/browse/FALCON-291
> Project: Falcon
> Issue Type: Bug
> Components: build-tools, oozie
>Affects Versions: 0.4
> Environment: HDP 1.3 and CDH 4.3
>Reporter: prashant
>
> Hi,
> I am not able to build falcon 0.4-incubating jar and getting error when i am
> trying to build it on HDP 1.3 as well as CDH 4.3.
> below are steps which i am following
> * git clone https://git-wip-us.apache.org/repos/asf/incubator-falcon.git
> falcon
> * cd falcon
> * export mvn package
> below is the error
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec
> (BUILD-OOZIE) on project build-tools: Command execution failed. Process
> exited with an error: 1 (Exit value: 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :build-tools
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FALCON-291) Not able to package Falcon
[
https://issues.apache.org/jira/browse/FALCON-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242649#comment-15242649
]
Ajay Yadava commented on FALCON-291:
Hi jp,
{quote}
curl: (6) Couldn't resolve host 'archive.apache.org'
{quote}
This seems to indicate that there was some problem accessing the above url.
Basically build is trying to download
http://archive.apache.org/dist/oozie/4.1.0/oozie-4.1.0.tar.gz but it fails.
Can you try accessing this url and let us know if you are able to open it?
> Not able to package Falcon
> --
>
> Key: FALCON-291
> URL: https://issues.apache.org/jira/browse/FALCON-291
> Project: Falcon
> Issue Type: Bug
> Components: build-tools, oozie
>Affects Versions: 0.4
> Environment: HDP 1.3 and CDH 4.3
>Reporter: prashant
>
> Hi,
> I am not able to build falcon 0.4-incubating jar and getting error when i am
> trying to build it on HDP 1.3 as well as CDH 4.3.
> below are steps which i am following
> * git clone https://git-wip-us.apache.org/repos/asf/incubator-falcon.git
> falcon
> * cd falcon
> * export mvn package
> below is the error
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec
> (BUILD-OOZIE) on project build-tools: Command execution failed. Process
> exited with an error: 1 (Exit value: 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :build-tools
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FALCON-1882) Instance status api not working via prism
[
https://issues.apache.org/jira/browse/FALCON-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Yadava resolved FALCON-1882.
-
Resolution: Fixed
Fix Version/s: 0.10
Issue resolved by pull request 87
[https://github.com/apache/falcon/pull/87]
> Instance status api not working via prism
> -
>
> Key: FALCON-1882
> URL: https://issues.apache.org/jira/browse/FALCON-1882
> Project: Falcon
> Issue Type: Bug
> Components: prism
>Affects Versions: trunk, 0.10
> Environment: QA
>Reporter: Pragya Mittal
>Assignee: Praveen Adlakha
> Fix For: 0.10
>
>
> {noformat}
> dataqa@lda01:falcon instance -type process -status -name
> EntitySummaryTest-agregator-coord16-19cf7cef4
> ERROR: Bad
> Request;ua1/org.apache.falcon.FalconException::org.apache.falcon.FalconException:
>
>
>
> Error 404 Not Found
>
> HTTP ERROR 404
> Problem accessing
> /api/instance/status/process/EntitySummaryTest-agregator-coord16-19cf7cef4.
> Reason:
> Not FoundPowered by
> Jetty://
> {noformat}
> Prism logs say :
> {noformat}
> 2016-04-04 12:52:54,822 ERROR - [1796371666@qtp-231977479-0 -
> 8f0a2c49-66c1-4cb1-b62d-4dcc62c947c4:dataqa:GET//instance/status/process/EntitySummaryTest-agregator-coord16-19cf7cef4]
> ~ Failed to fetch results for colo:ua1 (InstanceManagerProxy:629)
> org.apache.falcon.FalconException: org.apache.falcon.FalconException:
>
>
> Error 404 Not Found
>
> HTTP ERROR 404
> Problem accessing
> /api/instance/status/process/EntitySummaryTest-agregator-coord16-19cf7cef4.
> Reason:
> Not FoundPowered by
> Jetty://
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> at
> org.apache.falcon.resource.channel.HTTPChannel.invoke(HTTPChannel.java:134)
> at
> org.apache.falcon.resource.proxy.InstanceManagerProxy$3.doExecute(InstanceManagerProxy.java:237)
> at
> org.apache.falcon.resource.proxy.InstanceManagerProxy$3.doExecute(InstanceManagerProxy.java:1)
> at
> org.apache.falcon.resource.proxy.InstanceManagerProxy$InstanceProxy.execute(InstanceManagerProxy.java:623)
> at
> org.apache.falcon.resource.proxy.InstanceManagerProxy.getStatus_aroundBody4(InstanceManagerProxy.java:241)
> at
> org.apache.falcon.resource.proxy.InstanceManagerProxy$AjcClosure5.run(InstanceManagerProxy.java:1)
> at
> org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
> at
> org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
> at
> org.apache.falcon.resource.proxy.InstanceManagerProxy.getStatus(InstanceManagerProxy.java:220)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>
[jira] [Resolved] (FALCON-1845) Retries Stopped happening for all entities when one entity was deleted during rerun of instance
[ https://issues.apache.org/jira/browse/FALCON-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava resolved FALCON-1845. - Resolution: Fixed Fix Version/s: 0.10 Issue resolved by pull request 62 [https://github.com/apache/falcon/pull/62] > Retries Stopped happening for all entities when one entity was deleted > during rerun of instance > > > Key: FALCON-1845 > URL: https://issues.apache.org/jira/browse/FALCON-1845 > Project: Falcon > Issue Type: Bug > Components: rerun >Reporter: pavan kumar kolamuri >Assignee: pavan kumar kolamuri > Fix For: 0.10 > > > Whenever entity was deleted during rerun of instance of that entity , there > is chance that RetryHandler thread was stopped. Further retries won't happen > in FalconServer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FALCON-1866) Bug in JDBCStateStore
[
https://issues.apache.org/jira/browse/FALCON-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Yadava resolved FALCON-1866.
-
Resolution: Fixed
Fix Version/s: 0.10
Issue resolved by pull request 80
[https://github.com/apache/falcon/pull/80]
> Bug in JDBCStateStore
> -
>
> Key: FALCON-1866
> URL: https://issues.apache.org/jira/browse/FALCON-1866
> Project: Falcon
> Issue Type: Bug
>Reporter: Praveen Adlakha
>Assignee: Praveen Adlakha
> Fix For: 0.10
>
>
> Hi ,
> We have this piece of code :
> {code}
> if (result.isEmpty()) {
> return null;
> }
> entityManager.close();
> {code}
> so when ever the value is null entitymanager.close never gets executed.We
> should correct it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FALCON-1889) Falcon JMS created too many topics - one per process/feed
[ https://issues.apache.org/jira/browse/FALCON-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227649#comment-15227649 ] Ajay Yadava commented on FALCON-1889: - This may be helpful for you to look at - FALCON-1031 > Falcon JMS created too many topics - one per process/feed > - > > Key: FALCON-1889 > URL: https://issues.apache.org/jira/browse/FALCON-1889 > Project: Falcon > Issue Type: Bug > Components: general >Reporter: Venkatesan Ramachandran >Assignee: Venkatesan Ramachandran > > Per https://falcon.apache.org/Operability.html, Falcon creates a topic for > feed/process to publish messages. > If there are large number of feeds/processes, it would create too many JMS > topics that will be un-managable. If this is true, this needs to be fixed > and all the messages to be published to one well-know topic that external > applications can consume from. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[DISCUSS] Apache Falcon 1.0 release
Hello everyone, For quite some time we have been discussing to make a 1.0 release and have had several discussions in developer sync up around it. Taking this to next step, I propose next release line(after 0.10) of Apache Falcon to be 1.0 *Item 1 - Scope and Timelines* Some of the items that are in works and I personally feel will be idle candidate for 1.0 are - clean up our APIs(add a new version), introduce a new shell for Falcon feed sla alerts, and move to the more powerful and capable lifecycle framework for feeds among few. After lot of thought and discussions with several members, I propose to not aim for too many big features and a timeline of 2.5-3 months after the 0.10 release. This will ensure that critical fixes are not delayed and there is only one active working line for code. We can add more features if other community members are able to get them committed in time and our quality team also feels comfortable. *Item 2 - Migration Strategy* While some of the changes like REST API clean up etc. can be done by adding versioning others like migration to lifecycle framework etc. are bit more involved. One important decision to be made is how to migrate to 1.0 release. Here are some of the options to migrate to new entity definitions in 1.0 (NOTE: REST api can be versioned and same end points can continue to work albeit with newer definitions) *Approach 1*. Take a one time hit, call the release backward incompatible and provide changes inside falcon to automatically migrate to newer definitions on start up. We can support this migration code for a couple of releases and then later on remove it. pros: - Clean and easy to code - no if else etc. for supporting features in multiple manners. - Intuitive for users - multiple options for same purpose are confusing for the users. - Easier to maintain - All bug fixes need to be done only in one code flow and not at many places. cons: - involves migration - it can be automated by incorporating the migration as part of falcon startup. *Approach 2*. Support both old and new entity definitions. pros - users can work with both versions and migrate at their own pace cons - Hard to maintain - Lot of code will need to be duplicated (validations etc.) or branched (wf builders etc.) All bug fixes will need to be fixed in multiple places. - Not scalable approach - The code will not scale easily if we decide to support one more version. - Manual migration - Users will have to migrate themselves to the new entity definitions. - Gotchas - What will happen if someone submits in newer version but calls update in older version? I invite all of you to provide your thoughts on both the items. There might be more approaches or points to consider, suggestions are welcome. Cheers Ajay Yadava
[jira] [Updated] (FALCON-1866) Bug in JDBCStateStore
[
https://issues.apache.org/jira/browse/FALCON-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Yadava updated FALCON-1866:
Summary: Bug in JDBCStateStore (was: bug in JDBCStateStore)
> Bug in JDBCStateStore
> -
>
> Key: FALCON-1866
> URL: https://issues.apache.org/jira/browse/FALCON-1866
> Project: Falcon
> Issue Type: Bug
>Reporter: Praveen Adlakha
>Assignee: Praveen Adlakha
>
> Hi ,
> We have this piece of code :
> {code}
> if (result.isEmpty()) {
> return null;
> }
> entityManager.close();
> {code}
> so when ever the value is null entitymanager.close never gets executed.We
> should correct it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FALCON-1867) hardcoded query names in JDBCStateStore
[ https://issues.apache.org/jira/browse/FALCON-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215347#comment-15215347 ] Ajay Yadava commented on FALCON-1867: - I think the issue is that you can make typos and not figure it out till runtime as there is no compile time check that you actually typed the exact same string that you wanted to use. > hardcoded query names in JDBCStateStore > --- > > Key: FALCON-1867 > URL: https://issues.apache.org/jira/browse/FALCON-1867 > Project: Falcon > Issue Type: Bug >Reporter: Praveen Adlakha >Assignee: Praveen Adlakha > > Hi All, > All the query names in JDBC statestore are hardcoded string across two > different classes which is not a good practice and can causes run time errors. > Need to change that. > Thanks > Praveen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FALCON-1865) Persist Feed sla data to database
[ https://issues.apache.org/jira/browse/FALCON-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava resolved FALCON-1865. - Resolution: Fixed Fix Version/s: trunk Issue resolved by pull request 77 [https://github.com/apache/falcon/pull/77] > Persist Feed sla data to database > -- > > Key: FALCON-1865 > URL: https://issues.apache.org/jira/browse/FALCON-1865 > Project: Falcon > Issue Type: New Feature > Reporter: Ajay Yadava >Assignee: Praveen Adlakha > Fix For: trunk > > > Currently Feed SLA monitoring persists its state in HDFS. With native > scheduler we can leverage database to store the state and provide richer set > of features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FALCON-1873) Falcon export to Druid
Ajay Yadava created FALCON-1873: --- Summary: Falcon export to Druid Key: FALCON-1873 URL: https://issues.apache.org/jira/browse/FALCON-1873 Project: Falcon Issue Type: New Feature Reporter: Ajay Yadava Assignee: Ajay Yadava Fix For: 1.0 Support export to druid as export lifecycle stage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FALCON-1870) Archival lifecycle stage
[ https://issues.apache.org/jira/browse/FALCON-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207872#comment-15207872 ] Ajay Yadava commented on FALCON-1870: - Yes, that makes sense. I highlighted the use case (archival to S3) more than the actual feature, so changed the description accordingly. > Archival lifecycle stage > > > Key: FALCON-1870 > URL: https://issues.apache.org/jira/browse/FALCON-1870 > Project: Falcon > Issue Type: New Feature > Components: feed-lifecycle > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Labels: lifecycle > Fix For: 1.0 > > > Introduce another lifecycle stage - archival. One very common use case is to > support archival of data to S3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FALCON-1870) Archival lifecycle stage
[ https://issues.apache.org/jira/browse/FALCON-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadava updated FALCON-1870: Description: Introduce another lifecycle stage - archival. One very common use case is to support archival of data to S3. (was: Introduce another lifecycle stage - archival. Support archival of data to S3.) > Archival lifecycle stage > > > Key: FALCON-1870 > URL: https://issues.apache.org/jira/browse/FALCON-1870 > Project: Falcon > Issue Type: New Feature > Components: feed-lifecycle > Reporter: Ajay Yadava > Assignee: Ajay Yadava > Labels: lifecycle > Fix For: 1.0 > > > Introduce another lifecycle stage - archival. One very common use case is to > support archival of data to S3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
