Review Request 38448: Documentation for Falcon Email notification

2015-09-17 Thread Peeyush Bishnoi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38448/
---

Review request for Falcon.


Bugs: FALCON-1426
https://issues.apache.org/jira/browse/FALCON-1426


Repository: falcon-git


Description
---

FALCON-1426: Documentation for Falcon Email notification


Diffs
-

  docs/src/site/twiki/EntitySpecification.twiki bf45598 
  docs/src/site/twiki/FalconDocumentation.twiki 68ed2e2 
  docs/src/site/twiki/FalconEmailNotification.twiki PRE-CREATION 

Diff: https://reviews.apache.org/r/38448/diff/


Testing
---

Yes.


Thanks,

Peeyush Bishnoi

[jira] [Commented] (FALCON-965) Open up life cycle stage implementation within Falcon for extension

2015-09-17 Thread Ajay Yadava (JIRA) 

[ 
https://issues.apache.org/jira/browse/FALCON-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791742#comment-14791742
 ] 

Ajay Yadava commented on FALCON-965:


I have uploaded the patch to review board, one thing which is missing is the 
user documentation. I will finish that and add it soon.

> Open up life cycle stage implementation within Falcon for extension
> ---
>
> Key: FALCON-965
> URL: https://issues.apache.org/jira/browse/FALCON-965
> Project: Falcon
>  Issue Type: New Feature
>Affects Versions: 0.7
>Reporter: Srikanth Sundarrajan
>Assignee: Ajay Yadava
>  Labels: recipes
> Fix For: 0.8
>
> Attachments: FALCON-965.patch, FalconLifecycle-Designdoc.pdf, 
> xsd.patch
>
>
> As it stands Falcon supports replication, generation and eviction lifecycle 
> stages and plans to support more. This however assumes a certain way of 
> implementing a life cycle function and changes to these implementation aren't 
> easy, as they are not open for easy extension. This proposed feature is open 
> this up in Falcon.
> Here is a proposal on how things can possibly be:
> * List of life cycles that Falcon supports would be well known and not 
> extensible
> * Dependency between life cycles are coded up in the falcon server and not 
> necessarily extensible. (In short adding a new life cycle still requires 
> changes in Falcon)
> * Each Lifecycle in falcon advertises an implementation interface and minimum 
> configuration interface (for ex. Eviction should expose a way to retrieve the 
> configured time limit for which data will be available for other life cycle 
> stages to validate. There is no point in having a process consume last 24 
> instances of a feed, when the retention will retain only 4 instances)
> * Similar to FALCON-634, life cycle implementation can be dropped in as long 
> as the implementation interface and configuraion interfaces are adhered to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Blog post on Apache Falcon (and a few more tools)

2015-09-17 Thread Adam Kawa 
Hi Guys,

I have written the second part of this blog post series
http://getindata.com/blog/post/avoiding-the-mess-in-the-hadoop-cluster-part-2

Hope that you will find it interesting and useful for the community!

Cheers!
Adam

2015-05-26 2:29 GMT+02:00 Srikanth Sundarrajan :

> Blog is now published at
> https://blogs.apache.org/falcon/entry/avoiding_the_mess_in_the
>
> Regards
> Srikanth Sundarrajan
>
> > Subject: Re: Blog post on Apache Falcon (and a few more tools)
> > From: srik...@hotmail.com
> > Date: Tue, 28 Apr 2015 06:31:45 +0530
> > To: dev@falcon.apache.org
> >
> > It certainly is, will take care of it ASAP
> >
> > Sent from my iPhone
> >
> > > On 28-Apr-2015, at 2:27 am, "Adam Kawa"  wrote:
> > >
> > > Hi Srikanth,
> > >
> > > I hope you haven't changed your mind and still think that it's
> valuable to
> > > re-publish this blog post :)
> > >
> > > Cheers!
> > > Adam
> > >
> > > 2015-04-02 10:47 GMT+02:00 Adam Kawa :
> > >
> > >> It's fine to skip it. It doesn't introduce much value as it's not a
> > >> diagram or a real-world dashboard.
> > >>
> > >> 2015-04-01 17:58 GMT+02:00 Srikanth Sundarrajan  >:
> > >>
> > >>> 1. Just in case please remove the picture from the "Automatic data
> > >>> retention" section.
> > >>> <<
> > >>> Would you like to use something instead ?
> > >>>
> > >>> 
> >  Date: Wed, 1 Apr 2015 15:37:32 +0200
> >  Subject: Re: Blog post on Apache Falcon (and a few more tools)
> >  From: kawa.a...@gmail.com
> >  To: srik...@hotmail.com
> >  CC: dev@falcon.apache.org
> > 
> >  Hi Srikanth,
> > 
> >  1. Just in case please remove the picture from the "Automatic data
> >  retention" section.
> >  2. Other pictures are taken from http://pixabay.com/en/ e.g.
> >  http://pixabay.com/en/cowboy-sunset-lake-dusk-country-283449,
> >  http://pixabay.com/en/elephant-animal-mammal-trunk-happy-238106/ or
> > >>> from
> >  https://issues.apache.org/jira/browse/FALCON-790 or the Falcon
> server
> >  running on own cluster.
> > 
> >  Thanks!
> >  Adam
> > 
> >  2015-03-28 17:23 GMT+01:00 Srikanth Sundarrajan <
> srik...@hotmail.com>:
> > 
> > > Hi Adam,
> > > Here is how the blog will appear on the Apache blogs. Am not sure
> what
> > > are the copyright implications of the images used in the blog if
> they
> > >>> are
> > > reposted. For now, am referring to the images directly in getindata
> > >>> instead
> > > of hosting them at blogs.apache.org.
> > >
> > > Regards
> > > Srikanth Sundarrajan
> > > 
> > >> Date: Sat, 28 Mar 2015 09:43:17 +0100
> > >> Subject: Re: Blog post on Apache Falcon (and a few more tools)
> > >> From: kawa.a...@gmail.com
> > >> To: dev@falcon.apache.org
> > >>
> > >> Yes, that's fine!
> > >>
> > >> 2015-03-28 7:45 GMT+01:00 Srikanth Sundarrajan <
> srik...@hotmail.com>:
> > >>
> > >>> Hi Adam,
> > >>> I can take care reposting this in apache blog and also credit and
> > >>> refer
> > >>> to the original blog, but it will appear as being authored by
> the one
> > >>> actually posting the blog. Are you ok with that ?
> > >>>
> > >>> Regards
> > >>> Srikanth Sundarrajan
> > >>>
> > >>> 
> >  Date: Fri, 27 Mar 2015 20:26:52 +0100
> >  Subject: Re: Blog post on Apache Falcon (and a few more tools)
> >  From: kawa.a...@gmail.com
> >  To: dev@falcon.apache.org
> > 
> >  Cool! :)
> > 
> >  The second part will be written in a month or so... It will also
> > >>> cover
> > >>> some
> >  features of Falcon.
> > 
> >  2015-03-27 19:51 GMT+01:00 Seetharam Venkatesh <
> > > venkat...@innerzeal.com
> >  :
> > 
> > > Either me or Srikanth can take care of this. Thanks!
> > >
> > > On Thu, Mar 26, 2015 at 4:15 PM, Adam Kawa <
> kawa.a...@gmail.com>
> > > wrote:
> > >
> > >> 2015-03-25 22:00 GMT+01:00 Seetharam Venkatesh <
> > >>> venkat...@innerzeal.com
> > >> :
> > >>
> > >>> We should probably republish this in Apache as well if you
> desire
> > > so.
> > >>
> > >> This would be really great! How could we proceed?
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Venkatesh
> > >
> > > “Perfection (in design) is achieved not when there is nothing
> more
> > >>> to
> > >>> add,
> > > but rather when there is nothing more to take away.”
> > > - Antoine de Saint-Exupéry
> > >>
> > >>
>
>

[jira] [Updated] (FALCON-1426) Add documentation for Falcon Email notification

2015-09-17 Thread Peeyush Bishnoi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peeyush Bishnoi updated FALCON-1426:

Fix Version/s: 0.8

> Add documentation for Falcon Email notification
> ---
>
> Key: FALCON-1426
> URL: https://issues.apache.org/jira/browse/FALCON-1426
> Project: Falcon
>  Issue Type: Sub-task
>  Components: common
>Affects Versions: 0.7
>Reporter: Peeyush Bishnoi
>Assignee: Peeyush Bishnoi
> Fix For: 0.8
>
> Attachments: FALCON-1426.patch
>
>
> Add documentation how to use Falcon email notification. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1426) Add documentation for Falcon Email notification

2015-09-17 Thread Peeyush Bishnoi (JIRA) 

[ 
https://issues.apache.org/jira/browse/FALCON-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791683#comment-14791683
 ] 

Peeyush Bishnoi commented on FALCON-1426:
-

Documentation patch is attached that will illustrate how to use the Falcon 
email notification. Please review.

> Add documentation for Falcon Email notification
> ---
>
> Key: FALCON-1426
> URL: https://issues.apache.org/jira/browse/FALCON-1426
> Project: Falcon
>  Issue Type: Sub-task
>  Components: common
>Reporter: Peeyush Bishnoi
>Assignee: Peeyush Bishnoi
> Attachments: FALCON-1426.patch
>
>
> Add documentation how to use Falcon email notification. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1426) Add documentation for Falcon Email notification

2015-09-17 Thread Peeyush Bishnoi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peeyush Bishnoi updated FALCON-1426:

Attachment: FALCON-1426.patch

> Add documentation for Falcon Email notification
> ---
>
> Key: FALCON-1426
> URL: https://issues.apache.org/jira/browse/FALCON-1426
> Project: Falcon
>  Issue Type: Sub-task
>  Components: common
>Reporter: Peeyush Bishnoi
>Assignee: Peeyush Bishnoi
> Attachments: FALCON-1426.patch
>
>
> Add documentation how to use Falcon email notification. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1426) Add documentation for Falcon Email notification

2015-09-17 Thread Peeyush Bishnoi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peeyush Bishnoi updated FALCON-1426:

Component/s: (was: common)
 docs

> Add documentation for Falcon Email notification
> ---
>
> Key: FALCON-1426
> URL: https://issues.apache.org/jira/browse/FALCON-1426
> Project: Falcon
>  Issue Type: Sub-task
>  Components: docs
>Affects Versions: 0.7
>Reporter: Peeyush Bishnoi
>Assignee: Peeyush Bishnoi
> Fix For: 0.8
>
> Attachments: FALCON-1426.patch
>
>
> Add documentation how to use Falcon email notification. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1426) Add documentation for Falcon Email notification

2015-09-17 Thread Peeyush Bishnoi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peeyush Bishnoi updated FALCON-1426:

Affects Version/s: 0.7

> Add documentation for Falcon Email notification
> ---
>
> Key: FALCON-1426
> URL: https://issues.apache.org/jira/browse/FALCON-1426
> Project: Falcon
>  Issue Type: Sub-task
>  Components: common
>Affects Versions: 0.7
>Reporter: Peeyush Bishnoi
>Assignee: Peeyush Bishnoi
> Fix For: 0.8
>
> Attachments: FALCON-1426.patch
>
>
> Add documentation how to use Falcon email notification. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 38450: FALCON-965 Open up Feed lifecycle for user extension

2015-09-17 Thread Ajay Yadava 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38450/
---

Review request for Falcon.


Bugs: FALCON-965
https://issues.apache.org/jira/browse/FALCON-965


Repository: falcon-git


Description
---

For details on feed lifecycle feature and motivation behind it, please refer 
FALCON-965. This is the base framework for lifecycle with a feature parity of 
retention stage as a reference implementation.


Diffs
-

  client/src/main/resources/feed-0.1.xsd 4ff8baa 
  common/src/main/java/org/apache/falcon/entity/FeedHelper.java 572923b 
  
common/src/main/java/org/apache/falcon/entity/parser/CrossEntityValidations.java
 2696552 
  common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 
f22a343 
  
common/src/main/java/org/apache/falcon/lifecycle/AbstractPolicyBuilderFactory.java
 PRE-CREATION 
  common/src/main/java/org/apache/falcon/lifecycle/FeedLifecycleStage.java 
PRE-CREATION 
  common/src/main/java/org/apache/falcon/lifecycle/LifecyclePolicy.java 
PRE-CREATION 
  common/src/main/java/org/apache/falcon/lifecycle/PolicyBuilder.java 
PRE-CREATION 
  
common/src/main/java/org/apache/falcon/lifecycle/retention/NominalTimeBasedDelete.java
 PRE-CREATION 
  common/src/main/java/org/apache/falcon/service/LifecyclePolicyStore.java 
PRE-CREATION 
  common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 
756c6b8 
  common/src/main/resources/startup.properties 39a412d 
  common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java 6179855 
  common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java c70cfcc 
  
common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java 
754fb06 
  common/src/test/resources/config/feed/feed-0.3.xml PRE-CREATION 
  common/src/test/resources/config/feed/feed-0.4.xml PRE-CREATION 
  lifecycle/pom.xml PRE-CREATION 
  
lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/OoziePolicyBuilderFactory.java
 PRE-CREATION 
  
lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedCoordinatorBuilder.java
 PRE-CREATION 
  
lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedDeleteBuilder.java
 PRE-CREATION 
  
lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedWorkflowBuilder.java
 PRE-CREATION 
  
lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/utils/OozieBuilderUtils.java
 PRE-CREATION 
  lifecycle/src/main/resources/action/feed/eviction-action.xml PRE-CREATION 
  lifecycle/src/main/resources/binding/jaxb-binding.xjb PRE-CREATION 
  oozie/pom.xml 157edf9 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 
b819dee 
  
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
 7d0174a 
  oozie/src/test/resources/feed/fs-retention-feed.xml PRE-CREATION 
  oozie/src/test/resources/feed/fs-retention-lifecycle-feed.xml PRE-CREATION 
  pom.xml 646de69 
  src/conf/startup.properties 305ac36 

Diff: https://reviews.apache.org/r/38450/diff/


Testing
---

Unit tests have been added for all the methods.
I have compared the new artifacts(workflow, bundle, config-default and 
coordinator xmls) with the old ones.


Thanks,

Ajay Yadava

Review Request 38459: Make validity end date optional for feed / process

2015-09-17 Thread PRAGYA MITTAL 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38459/
---

Review request for Falcon and Ajay Yadava.


Bugs: FALCON-1417
https://issues.apache.org/jira/browse/FALCON-1417


Repository: falcon-git


Description
---

Added optional date functionality.


Diffs
-

  
common/src/main/java/org/apache/falcon/entity/parser/CrossEntityValidations.java
 2696552 
  common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 
992fc51 
  common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java 
48a4286 
  common/src/main/java/org/apache/falcon/util/DateUtil.java e736340 
  
common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java 
d203b7c 
  
common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java
 77f6a77 

Diff: https://reviews.apache.org/r/38459/diff/


Testing
---

Added test cases for the same.


Thanks,

PRAGYA MITTAL

[jira] [Updated] (FALCON-1417) Make validity end date optional for feed / process

2015-09-17 Thread Ajay Yadava (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Yadava updated FALCON-1417:

Fix Version/s: 0.8

> Make validity end date optional for feed / process
> --
>
> Key: FALCON-1417
> URL: https://issues.apache.org/jira/browse/FALCON-1417
> Project: Falcon
>  Issue Type: Improvement
>Reporter: Ajay Yadava
>Assignee: Pragya Mittal
> Fix For: 0.8
>
> Attachments: FALCON-1417.patch
>
>
> Currently in most scenarios when user submit feed/process they don't want 
> have a specified end date in their mind, hence most users end up specifying a 
> random large end date like in year 2999 etc. We can make end date optional 
> and solve for such use cases.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Scoping Apache Falcon 0.8 release

2015-09-17 Thread Ajay Yadav 
Hi Sowmya,

I have migrated all the issues that I committed in master to 0.8 and did a
quick check using CHANGES.txt to see if all issues committed have been
moved to 0.8.
Depending on the feature freeze date, following are the major features I am
working on and would like to propose for 0.8.

   1. Feed lifecycle (FALCON-965)
   2. Feed SLA monitoring (FALCON-436)
   3. Process SLA monitoring (FALCON-1090)



Cheers
Ajay Yadava

On Tue, Sep 15, 2015 at 4:38 PM, Srikanth Sundarrajan 
wrote:

> Hi Sowmya, Thanks much for driving the next release for Falcon. I have
> created 0.8 version on the JIRA system and this should allow us to tag
> JIRAs with 0.8 if they are to be scoped in the upcoming release.
>
> Regards
> Srikanth Sundarrajan
>
> > Subject: Scoping Apache Falcon 0.8 release
> > From: sram...@hortonworks.com
> > To: dev@falcon.apache.org
> > Date: Tue, 15 Sep 2015 01:21:09 +
> >
> > Hello everyone,
> >
> > We are planning to do another release of Apache Falcon around second
> week of October. I would like to invite everyone to help in freezing the
> scope of 0.8 release.
> > I have opened JIRA  https://issues.apache.org/jira/browse/FALCON-1450
> to cover the tasks under this release.
> >
> > We would like to include all or as many of "Patch Available" JIRA's as
> long as the they are ready to be committed shortly and don't move the
> release timeline.
> >
> > I request everybody to provide with suggestion for JIRA's to be added in
> scope for 0.8 release. We want to freeze the scope of release and cut
> branch for 0.8 before the weekend, so if you have any important JIRA
> > which you would want to be included in 0.8 then please let me know at
> the earliest.
> >
> > Feel free to comment on the JIRA if you have any comments/suggestions.
> >
> > Regards,
> > Sowmya Ramesh
>
>

-- 
_
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.

[jira] [Updated] (FALCON-1425) Provide Email based plugin to send Notification once instance completed

2015-09-17 Thread Peeyush Bishnoi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peeyush Bishnoi updated FALCON-1425:

Attachment: FALCON-1425.v4.patch

> Provide Email based plugin to send Notification once instance completed
> ---
>
> Key: FALCON-1425
> URL: https://issues.apache.org/jira/browse/FALCON-1425
> Project: Falcon
>  Issue Type: Sub-task
>  Components: common
>Affects Versions: 0.7
>Reporter: Peeyush Bishnoi
>Assignee: Peeyush Bishnoi
> Fix For: 0.8
>
> Attachments: FALCON-1425.patch, FALCON-1425.v1.patch, 
> FALCON-1425.v2.patch, FALCON-1425.v3.patch, FALCON-1425.v4.patch
>
>
> Requirement is to provide Email based notification plugin to send 
> notification when Falcon instance completes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38448: Documentation for Falcon Email notification

2015-09-17 Thread Scott Preece 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38448/#review99390
---



docs/src/site/twiki/FalconEmailNotification.twiki (line 3)


Recommend text change:
Change "sending email based notification from Falcon when scheduled 
feed/process instance completed"

To:
"sending email notifications when scheduled feed/process instances 
complete."



docs/src/site/twiki/FalconEmailNotification.twiki (line 8)


Will it confuse users that the "to" part looks like a literal e-mail 
address? Will people think that's the value they should use, rather than the 
intended recipient address?



docs/src/site/twiki/FalconEmailNotification.twiki (line 14)


Recommend a text change:
"specifies the address to send notifications to; multiple recipients may be 
provided as a comma-separated list"


- Scott Preece


On Sept. 17, 2015, 7:07 a.m., Peeyush Bishnoi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38448/
> ---
> 
> (Updated Sept. 17, 2015, 7:07 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1426
> https://issues.apache.org/jira/browse/FALCON-1426
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> FALCON-1426: Documentation for Falcon Email notification
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/EntitySpecification.twiki bf45598 
>   docs/src/site/twiki/FalconDocumentation.twiki 68ed2e2 
>   docs/src/site/twiki/FalconEmailNotification.twiki PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38448/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> 
> Thanks,
> 
> Peeyush Bishnoi
> 
>

[jira] [Created] (FALCON-1463) UI - unable to create feed with empty schema location

2015-09-17 Thread JIRA 
Michał Ziemba created FALCON-1463:
-

 Summary: UI - unable to create feed with empty schema location
 Key: FALCON-1463
 URL: https://issues.apache.org/jira/browse/FALCON-1463
 Project: Falcon
  Issue Type: Bug
  Components: falcon-ui
Affects Versions: 0.7
Reporter: Michał Ziemba


Using the new UI, I'm unable to create a feed that has empty location attribute 
in schema tag:



(example of such feed is given here: 
http://falcon.apache.org/HiveIntegration.html)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38459: Make validity end date optional for feed / process

2015-09-17 Thread Balu Vellanki 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38459/#review99388
---



common/src/main/java/org/apache/falcon/util/DateUtil.java (line 40)


JavaDoc command does not pass checkstyle.



common/src/main/java/org/apache/falcon/util/DateUtil.java (line 43)


Is enum preferred over a static method?


- Balu Vellanki


On Sept. 17, 2015, 11:01 a.m., PRAGYA MITTAL wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38459/
> ---
> 
> (Updated Sept. 17, 2015, 11:01 a.m.)
> 
> 
> Review request for Falcon and Ajay Yadava.
> 
> 
> Bugs: FALCON-1417
> https://issues.apache.org/jira/browse/FALCON-1417
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> Added optional date functionality.
> 
> 
> Diffs
> -
> 
>   
> common/src/main/java/org/apache/falcon/entity/parser/CrossEntityValidations.java
>  2696552 
>   common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 
> 992fc51 
>   
> common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java 
> 48a4286 
>   common/src/main/java/org/apache/falcon/util/DateUtil.java e736340 
>   
> common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java
>  d203b7c 
>   
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java
>  77f6a77 
> 
> Diff: https://reviews.apache.org/r/38459/diff/
> 
> 
> Testing
> ---
> 
> Added test cases for the same.
> 
> 
> Thanks,
> 
> PRAGYA MITTAL
> 
>

[jira] [Commented] (FALCON-1401) MetadataMappingService fails to add an edge for a process instance

2015-09-17 Thread Balu Vellanki (JIRA) 

[ 
https://issues.apache.org/jira/browse/FALCON-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803221#comment-14803221
 ] 

Balu Vellanki commented on FALCON-1401:
---

[~pallavi.rao] : Hi Pallavi, can we get the fix in for 0.8? If yes, please 
update the fix version. We are seeing this issue as well, so do you have an ETA 
for the Jira?

> MetadataMappingService fails to add an edge for a process instance
> --
>
> Key: FALCON-1401
> URL: https://issues.apache.org/jira/browse/FALCON-1401
> Project: Falcon
>  Issue Type: Bug
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
>Priority: Minor
> Attachments: FALCON-1401.patch
>
>
> If a process has no output feeds, the metadata mapping service fails to add 
> an edge for the process instances. It fails with NPE:
> {code}
> 2015-08-14 15:31:00,950 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:00,951 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,957 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:00,981 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,984 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,988 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,990 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,992 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,995 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,997 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,009 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:01,009 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,012 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:01,033 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,035 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,039 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,041 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,043 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,044 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,047 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,073 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:01,073 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,076 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:01,096 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
>

Re: Blog post on Apache Falcon (and a few more tools)

2015-09-17 Thread Seetharam Venkatesh 
Sure, either me or Srikanth will post it, thanks! I did retweet this.

On Wed, Sep 16, 2015 at 11:57 PM Adam Kawa  wrote:

> Hi Guys,
>
> I have written the second part of this blog post series
>
> http://getindata.com/blog/post/avoiding-the-mess-in-the-hadoop-cluster-part-2
>
> Hope that you will find it interesting and useful for the community!
>
> Cheers!
> Adam
>
> 2015-05-26 2:29 GMT+02:00 Srikanth Sundarrajan :
>
> > Blog is now published at
> > https://blogs.apache.org/falcon/entry/avoiding_the_mess_in_the
> >
> > Regards
> > Srikanth Sundarrajan
> >
> > > Subject: Re: Blog post on Apache Falcon (and a few more tools)
> > > From: srik...@hotmail.com
> > > Date: Tue, 28 Apr 2015 06:31:45 +0530
> > > To: dev@falcon.apache.org
> > >
> > > It certainly is, will take care of it ASAP
> > >
> > > Sent from my iPhone
> > >
> > > > On 28-Apr-2015, at 2:27 am, "Adam Kawa"  wrote:
> > > >
> > > > Hi Srikanth,
> > > >
> > > > I hope you haven't changed your mind and still think that it's
> > valuable to
> > > > re-publish this blog post :)
> > > >
> > > > Cheers!
> > > > Adam
> > > >
> > > > 2015-04-02 10:47 GMT+02:00 Adam Kawa :
> > > >
> > > >> It's fine to skip it. It doesn't introduce much value as it's not a
> > > >> diagram or a real-world dashboard.
> > > >>
> > > >> 2015-04-01 17:58 GMT+02:00 Srikanth Sundarrajan <
> srik...@hotmail.com
> > >:
> > > >>
> > > >>> 1. Just in case please remove the picture from the "Automatic data
> > > >>> retention" section.
> > > >>> <<
> > > >>> Would you like to use something instead ?
> > > >>>
> > > >>> 
> > >  Date: Wed, 1 Apr 2015 15:37:32 +0200
> > >  Subject: Re: Blog post on Apache Falcon (and a few more tools)
> > >  From: kawa.a...@gmail.com
> > >  To: srik...@hotmail.com
> > >  CC: dev@falcon.apache.org
> > > 
> > >  Hi Srikanth,
> > > 
> > >  1. Just in case please remove the picture from the "Automatic data
> > >  retention" section.
> > >  2. Other pictures are taken from http://pixabay.com/en/ e.g.
> > >  http://pixabay.com/en/cowboy-sunset-lake-dusk-country-283449,
> > >  http://pixabay.com/en/elephant-animal-mammal-trunk-happy-238106/
> or
> > > >>> from
> > >  https://issues.apache.org/jira/browse/FALCON-790 or the Falcon
> > server
> > >  running on own cluster.
> > > 
> > >  Thanks!
> > >  Adam
> > > 
> > >  2015-03-28 17:23 GMT+01:00 Srikanth Sundarrajan <
> > srik...@hotmail.com>:
> > > 
> > > > Hi Adam,
> > > > Here is how the blog will appear on the Apache blogs. Am not sure
> > what
> > > > are the copyright implications of the images used in the blog if
> > they
> > > >>> are
> > > > reposted. For now, am referring to the images directly in
> getindata
> > > >>> instead
> > > > of hosting them at blogs.apache.org.
> > > >
> > > > Regards
> > > > Srikanth Sundarrajan
> > > > 
> > > >> Date: Sat, 28 Mar 2015 09:43:17 +0100
> > > >> Subject: Re: Blog post on Apache Falcon (and a few more tools)
> > > >> From: kawa.a...@gmail.com
> > > >> To: dev@falcon.apache.org
> > > >>
> > > >> Yes, that's fine!
> > > >>
> > > >> 2015-03-28 7:45 GMT+01:00 Srikanth Sundarrajan <
> > srik...@hotmail.com>:
> > > >>
> > > >>> Hi Adam,
> > > >>> I can take care reposting this in apache blog and also credit
> and
> > > >>> refer
> > > >>> to the original blog, but it will appear as being authored by
> > the one
> > > >>> actually posting the blog. Are you ok with that ?
> > > >>>
> > > >>> Regards
> > > >>> Srikanth Sundarrajan
> > > >>>
> > > >>> 
> > >  Date: Fri, 27 Mar 2015 20:26:52 +0100
> > >  Subject: Re: Blog post on Apache Falcon (and a few more tools)
> > >  From: kawa.a...@gmail.com
> > >  To: dev@falcon.apache.org
> > > 
> > >  Cool! :)
> > > 
> > >  The second part will be written in a month or so... It will
> also
> > > >>> cover
> > > >>> some
> > >  features of Falcon.
> > > 
> > >  2015-03-27 19:51 GMT+01:00 Seetharam Venkatesh <
> > > > venkat...@innerzeal.com
> > >  :
> > > 
> > > > Either me or Srikanth can take care of this. Thanks!
> > > >
> > > > On Thu, Mar 26, 2015 at 4:15 PM, Adam Kawa <
> > kawa.a...@gmail.com>
> > > > wrote:
> > > >
> > > >> 2015-03-25 22:00 GMT+01:00 Seetharam Venkatesh <
> > > >>> venkat...@innerzeal.com
> > > >> :
> > > >>
> > > >>> We should probably republish this in Apache as well if you
> > desire
> > > > so.
> > > >>
> > > >> This would be really great! How could we proceed?
> > > >
> > > >
> > > >
>

Re: Review Request 38450: FALCON-965 Open up Feed lifecycle for user extension

2015-09-17 Thread pavan kumar kolamuri 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38450/#review99371
---



common/src/main/java/org/apache/falcon/entity/FeedHelper.java (line 430)


Only one policy is returned. Something might missed here ?



common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 
(line 125)


can be moved out of forloop



common/src/main/java/org/apache/falcon/service/LifecyclePolicyStore.java (line 
69)


minor nit can we clear the policyMap ?



lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/OoziePolicyBuilderFactory.java
 (line 37)


Should this static block in AbstractPolicyBuilderFactory as we are loading 
Policies irrespective of builderfactory right ? Or this is specific to 
OoziePolicyBuilderFactory ?



pom.xml (line 851)


space


- pavan kumar kolamuri


On Sept. 17, 2015, 7:52 a.m., Ajay Yadava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38450/
> ---
> 
> (Updated Sept. 17, 2015, 7:52 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-965
> https://issues.apache.org/jira/browse/FALCON-965
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> For details on feed lifecycle feature and motivation behind it, please refer 
> FALCON-965. This is the base framework for lifecycle with a feature parity of 
> retention stage as a reference implementation.
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/feed-0.1.xsd 4ff8baa 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java 572923b 
>   
> common/src/main/java/org/apache/falcon/entity/parser/CrossEntityValidations.java
>  2696552 
>   common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 
> f22a343 
>   
> common/src/main/java/org/apache/falcon/lifecycle/AbstractPolicyBuilderFactory.java
>  PRE-CREATION 
>   common/src/main/java/org/apache/falcon/lifecycle/FeedLifecycleStage.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/falcon/lifecycle/LifecyclePolicy.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/falcon/lifecycle/PolicyBuilder.java 
> PRE-CREATION 
>   
> common/src/main/java/org/apache/falcon/lifecycle/retention/NominalTimeBasedDelete.java
>  PRE-CREATION 
>   common/src/main/java/org/apache/falcon/service/LifecyclePolicyStore.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 
> 756c6b8 
>   common/src/main/resources/startup.properties 39a412d 
>   common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java 6179855 
>   common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java c70cfcc 
>   
> common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java
>  754fb06 
>   common/src/test/resources/config/feed/feed-0.3.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-0.4.xml PRE-CREATION 
>   lifecycle/pom.xml PRE-CREATION 
>   
> lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/OoziePolicyBuilderFactory.java
>  PRE-CREATION 
>   
> lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedCoordinatorBuilder.java
>  PRE-CREATION 
>   
> lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedDeleteBuilder.java
>  PRE-CREATION 
>   
> lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedWorkflowBuilder.java
>  PRE-CREATION 
>   
> lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/utils/OozieBuilderUtils.java
>  PRE-CREATION 
>   lifecycle/src/main/resources/action/feed/eviction-action.xml PRE-CREATION 
>   lifecycle/src/main/resources/binding/jaxb-binding.xjb PRE-CREATION 
>   oozie/pom.xml 157edf9 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 
> b819dee 
>   
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
>  7d0174a 
>   oozie/src/test/resources/feed/fs-retention-feed.xml PRE-CREATION 
>   oozie/src/test/resources/feed/fs-retention-lifecycle-feed.xml PRE-CREATION 
>   pom.xml 646de69 
>   src/conf/startup.properties 305ac36 
> 
> Diff: https://reviews.apache.org/r/38450/diff/
> 
> 
> Testing
> ---
> 
> Unit tests have been added for all the methods.
> I have compared the new artifacts(workflow, bundle, config-default and 
> coordinator xmls) with the old ones.
> 
> 
> Thanks,
> 
> Ajay Yadava
> 
>

[jira] [Commented] (FALCON-1459) Ability to import from database

2015-09-17 Thread Venkat Ramachandran (JIRA) 

[ 
https://issues.apache.org/jira/browse/FALCON-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804339#comment-14804339
 ] 

Venkat Ramachandran commented on FALCON-1459:
-

Review Board - https://reviews.apache.org/r/38465/

> Ability to import from database
> ---
>
> Key: FALCON-1459
> URL: https://issues.apache.org/jira/browse/FALCON-1459
> Project: Falcon
>  Issue Type: Sub-task
>  Components: acquisition
>Affects Versions: trunk
>Reporter: Venkat Ramachandran
>Assignee: Venkat Ramachandran
> Attachments: FALCON-1459.20150916.patch
>
>
> Tracks the feature to import from database.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1027) Falcon REST API trusted proxy support

2015-09-17 Thread Sowmya Ramesh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sowmya Ramesh updated FALCON-1027:
--
Fix Version/s: (was: trunk)
   0.8

> Falcon REST API trusted proxy support
> -
>
> Key: FALCON-1027
> URL: https://issues.apache.org/jira/browse/FALCON-1027
> Project: Falcon
>  Issue Type: New Feature
>Affects Versions: 0.7
>Reporter: kenneth ho
>Assignee: Sowmya Ramesh
> Fix For: 0.8
>
> Attachments: ApacheFalcon-Proxyusersupport.pdf, FALCON-1027.V3.patch, 
> FALCON-1027.v0.patch, FALCON-1027.v1.patch, FALCON-1027.v2.patch
>
>
> In order for Falcon REST API to work securely via the Knox gateway it must be 
> possible to setup a trust relationship between Knox and Falcon. This is 
> commonly done in other Hadoop ecosystem components using a combination of 
> Kerberos/SPNego and a doas URL query parameter. This provides a mechanism for 
> Falcon to strongly authenticate Knox as a trusted proxy, ensuring that it can 
> trust the identity assertions made via the doas query parameter. The links 
> below provide some information describing how this is done for core Hadoop. 
> Also note that most components utilize Hadoop core's reusable hadoop-auth 
> module to implement this functionality.
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Proxy_Users
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Proxy_user



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Scoping Apache Falcon 0.8 release

2015-09-17 Thread Sowmya Ramesh 
Thanks Ajay, appreciate it!

All - code freeze date for 0.8 release is 5th October after which we won¹t
be considering any more code push for 0.8.
Please keep this timeline in mind for all the open JIRA¹s you want to be
included for 0.8 release.

Thanks!

On 9/17/15, 4:32 AM, "Ajay Yadav"  wrote:

>Hi Sowmya,
>
>I have migrated all the issues that I committed in master to 0.8 and did a
>quick check using CHANGES.txt to see if all issues committed have been
>moved to 0.8.
>Depending on the feature freeze date, following are the major features I
>am
>working on and would like to propose for 0.8.
>
>   1. Feed lifecycle (FALCON-965)
>   2. Feed SLA monitoring (FALCON-436)
>   3. Process SLA monitoring (FALCON-1090)
>
>
>
>Cheers
>Ajay Yadava
>
>On Tue, Sep 15, 2015 at 4:38 PM, Srikanth Sundarrajan
>
>wrote:
>
>> Hi Sowmya, Thanks much for driving the next release for Falcon. I have
>> created 0.8 version on the JIRA system and this should allow us to tag
>> JIRAs with 0.8 if they are to be scoped in the upcoming release.
>>
>> Regards
>> Srikanth Sundarrajan
>>
>> > Subject: Scoping Apache Falcon 0.8 release
>> > From: sram...@hortonworks.com
>> > To: dev@falcon.apache.org
>> > Date: Tue, 15 Sep 2015 01:21:09 +
>> >
>> > Hello everyone,
>> >
>> > We are planning to do another release of Apache Falcon around second
>> week of October. I would like to invite everyone to help in freezing the
>> scope of 0.8 release.
>> > I have opened JIRA  https://issues.apache.org/jira/browse/FALCON-1450
>> to cover the tasks under this release.
>> >
>> > We would like to include all or as many of "Patch Available" JIRA's as
>> long as the they are ready to be committed shortly and don't move the
>> release timeline.
>> >
>> > I request everybody to provide with suggestion for JIRA's to be added
>>in
>> scope for 0.8 release. We want to freeze the scope of release and cut
>> branch for 0.8 before the weekend, so if you have any important JIRA
>> > which you would want to be included in 0.8 then please let me know at
>> the earliest.
>> >
>> > Feel free to comment on the JIRA if you have any comments/suggestions.
>> >
>> > Regards,
>> > Sowmya Ramesh
>>
>>
>
>-- 
>_
>The information contained in this communication is intended solely for
>the 
>use of the individual or entity to whom it is addressed and others
>authorized to receive it. It may contain confidential or legally
>privileged 
>information. If you are not the intended recipient you are hereby
>notified 
>that any disclosure, copying, distribution or taking any action in
>reliance 
>on the contents of this information is strictly prohibited and may be
>unlawful. If you have received this communication in error, please notify
>us immediately by responding to this email and then delete it from your
>system. The firm is neither liable for the proper and complete
>transmission 
>of the information contained in this communication nor for any delay in
>its 
>receipt.

[jira] [Updated] (FALCON-1446) Flaky TaskLogRetrieverYarnTest

2015-09-17 Thread Sowmya Ramesh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sowmya Ramesh updated FALCON-1446:
--
Fix Version/s: 0.8

> Flaky TaskLogRetrieverYarnTest
> --
>
> Key: FALCON-1446
> URL: https://issues.apache.org/jira/browse/FALCON-1446
> Project: Falcon
>  Issue Type: Bug
>Reporter: Ajay Yadava
>Assignee: Narayan Periwal
> Fix For: 0.8
>
> Attachments: FALCON-1446-v0.patch
>
>
> Currently the build fails arbitrarily. 
> An example failure is below.
>  java.lang.AssertionError: null: lists don't have the same size expected:<2> 
> but was:<9>
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.failNotEquals(Assert.java:489)
> at org.testng.Assert.assertEquals(Assert.java:118)
> at org.testng.Assert.assertEquals(Assert.java:365)
> at org.testng.Assert.assertEquals(Assert.java:532)
> at org.testng.Assert.assertEquals(Assert.java:509)
> at 
> org.apache.falcon.oozie.logging.TaskLogRetrieverYarnTest.testSuccess(TaskLogRetrieverYarnTest.java:90)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1445) Docfix for api/instance/dependencies endpoint

2015-09-17 Thread Sowmya Ramesh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FALCON-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sowmya Ramesh updated FALCON-1445:
--
Fix Version/s: (was: trunk)
   0.8

> Docfix for api/instance/dependencies endpoint
> -
>
> Key: FALCON-1445
> URL: https://issues.apache.org/jira/browse/FALCON-1445
> Project: Falcon
>  Issue Type: Bug
>  Components: docs
>Affects Versions: 0.7
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 0.8
>
> Attachments: falcon-1445.patch
>
>
> The documentation mentions the endpoint to be:
> /api/instance/dependency/:entity-type/:entity-name
> The correct endpoint is:
> /api/instance/dependencies/:entity-type/:entity-name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 38465: FALCON-1459 : Ability to import from database

2015-09-17 Thread Venkatesan Ramachandran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38465/
---

Review request for Falcon, Ajay Yadava, Balu Vellanki, and Sowmya Ramesh.


Repository: falcon-git


Description
---

FALCON-1459 : Ability to import from database


Diffs
-

  client/src/main/java/org/apache/falcon/LifeCycle.java 58a2a6c 
  client/src/main/java/org/apache/falcon/Tag.java beeb812 
  client/src/main/java/org/apache/falcon/entity/v0/EntityType.java 0657124 
  client/src/main/java/org/apache/falcon/metadata/RelationshipType.java f034772 
  client/src/main/resources/datasource-0.1.xsd PRE-CREATION 
  client/src/main/resources/feed-0.1.xsd 4ff8baa 
  client/src/main/resources/jaxb-binding.xjb f644f40 
  client/src/main/resources/mysql_database.xml PRE-CREATION 
  common/src/main/java/org/apache/falcon/entity/DatasourceHelper.java 
PRE-CREATION 
  common/src/main/java/org/apache/falcon/entity/EntityUtil.java 2f05b1f 
  common/src/main/java/org/apache/falcon/entity/FeedHelper.java 572923b 
  
common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java
 PRE-CREATION 
  common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java 
5a33201 
  common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 
992fc51 
  common/src/main/java/org/apache/falcon/entity/store/ConfigurationStore.java 
e27187b 
  common/src/main/java/org/apache/falcon/entity/v0/EntityGraph.java bd4c6cf 
  common/src/main/java/org/apache/falcon/entity/v0/EntityIntegrityChecker.java 
bd32852 
  
common/src/main/java/org/apache/falcon/metadata/EntityRelationshipGraphBuilder.java
 8c3876c 
  common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java PRE-CREATION 
  common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 
4454239 
  common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java 6179855 
  common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java 640e87d 
  common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java c70cfcc 
  
common/src/test/java/org/apache/falcon/entity/parser/DatasourceEntityParserTest.java
 PRE-CREATION 
  
common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java 
d203b7c 
  common/src/test/java/org/apache/falcon/entity/v0/EntityGraphTest.java 3863b11 
  common/src/test/resources/config/datasource/datasource-0.1.xml PRE-CREATION 
  common/src/test/resources/config/datasource/datasource-invalid-0.1.xml 
PRE-CREATION 
  common/src/test/resources/config/feed/feed-import-0.1.xml PRE-CREATION 
  common/src/test/resources/config/feed/feed-import-invalid-0.1.xml 
PRE-CREATION 
  
falcon-regression/merlin-core/src/main/java/org/apache/falcon/regression/core/util/HiveAssert.java
 2a934b5 
  
oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java 
PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java 
PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java 
PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 
85f5330 
  
oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
 3213a70 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 
b819dee 
  oozie/src/main/resources/action/feed/import-sqoop-database-action.xml 
PRE-CREATION 
  pom.xml 646de69 
  retention/src/test/java/org/apache/falcon/retention/FeedEvictorTest.java 
72447da 
  webapp/pom.xml 828f7f5 
  webapp/src/test/java/org/apache/falcon/lifecycle/FeedImportIT.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java 
220e5a7 
  webapp/src/test/java/org/apache/falcon/resource/TestContext.java f031137 
  webapp/src/test/java/org/apache/falcon/util/HsqldbTestUtils.java PRE-CREATION 
  webapp/src/test/resources/datasource-template.xml PRE-CREATION 
  webapp/src/test/resources/feed-template3.xml PRE-CREATION 

Diff: https://reviews.apache.org/r/38465/diff/


Testing
---

* Unit tests
* Integration tests
* Manual tests
  * Setup MySQL, create table and populate
  * Create datasource and feed entity with import policy in Falcon  
  * Made sure the data lands up in the HDFS.


Thanks,

Venkatesan Ramachandran

Re: Review Request 35724: Base framework of the native scheduler

2015-09-17 Thread Venkatesan Ramachandran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35724/#review99479
---



common/src/main/java/org/apache/falcon/entity/EntityUtil.java (line 62)


Java date and calendar APIs have bugs (specifically related to day light 
savings) and performance issues. 

Please consider using joda time library (its Apache 2.0 license compatible).


- Venkatesan Ramachandran


On July 28, 2015, 11:07 a.m., Pallavi Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35724/
> ---
> 
> (Updated July 28, 2015, 11:07 a.m.)
> 
> 
> Review request for Falcon and Srikanth Sundarrajan.
> 
> 
> Bugs: FALCON-1213
> https://issues.apache.org/jira/browse/FALCON-1213
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> The patch has the basic framework for the scheduler. Each of the individual 
> service needs to be implemented completely and will be done as separate 
> JIRAs. The intention of the patch is ensure the base framework satisfies all 
> use cases and get any early feedback in terms of course correction.
> 
> 
> Diffs
> -
> 
>   common/pom.xml 36de1f5 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java b86d9d7 
>   
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java
>  c4f8843 
>   common/src/test/java/org/apache/falcon/entity/EntityUtilTest.java cfdc84d 
>   pom.xml 31997e8 
>   scheduler/pom.xml PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/exception/DAGEngineException.java 
> PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/exception/InvalidStateTransitionException.java
>  PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/exception/ServiceException.java 
> PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/exception/StateStoreException.java 
> PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java 
> PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 
> PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/execution/FalconExecutionService.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/execution/NotificationHandler.java 
> PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java
>  PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java 
> PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/FalconNotificationService.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/ServicesRegistry.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/event/DataEvent.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/event/Event.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/event/JobCompletedEvent.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/event/JobScheduledEvent.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/event/TimeElapsedEvent.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/DataNotificationService.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/JobCompletionService.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/TimeNotificationService.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/request/DataNotificationRequest.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/request/JobCompletionNotificationRequest.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/request/JobScheduleNotificationRequest.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/request/NotificationRequest.java
>  PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/notification/service/request/TimeNotificationRequest.java
>  PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 
> PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/state/EntityState.java 
> PRE-CREATION 
>   
> scheduler/src/main/java/org/apache/falcon/state/EntityStateChangeHandler.java 
> PRE-CREATION 
>

[jira] [Commented] (FALCON-1463) UI - unable to create feed with empty schema location

2015-09-17 Thread Balu Vellanki (JIRA) 

[ 
https://issues.apache.org/jira/browse/FALCON-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804951#comment-14804951
 ] 

Balu Vellanki commented on FALCON-1463:
---

[~ajayyadava] : I will look into this.

> UI - unable to create feed with empty schema location
> -
>
> Key: FALCON-1463
> URL: https://issues.apache.org/jira/browse/FALCON-1463
> Project: Falcon
>  Issue Type: Bug
>  Components: falcon-ui
>Affects Versions: 0.7
>Reporter: Michał Ziemba
>Assignee: Balu Vellanki
>
> Using the new UI, I'm unable to create a feed that has empty location 
> attribute in schema tag:
> 
> (example of such feed is given here: 
> http://falcon.apache.org/HiveIntegration.html)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38459: Make validity end date optional for feed / process

2015-09-17 Thread pavan kumar kolamuri 


> On Sept. 17, 2015, 12:20 p.m., pavan kumar kolamuri wrote:
> > common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java,
> >  line 85
> > 
> >
> > Before setting process end time shouldn't we check feed end times of 
> > this process ? Feed end time might be less than this right ?
> 
> Balu Vellanki wrote:
> There are two ways to handle the scenario where a process is being 
> submitted with it's feed having an endtime.
> 1) Get feed's endtime and use it for process. 
> or 
> 2) Set the end to DateUtil.OPTIONAL_DATE.NEVER.getDate() and let process 
> validation throw an exception.
> 
> I think that Falcon should not make assumptions about user intent in 
> option (1), so prefer option 2.

I agree balu, but there is chance that it will always throw exception, if the 
feed end time is set and this feature will never come into useful.


- pavan kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38459/#review99373
---


On Sept. 17, 2015, 11:01 a.m., PRAGYA MITTAL wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38459/
> ---
> 
> (Updated Sept. 17, 2015, 11:01 a.m.)
> 
> 
> Review request for Falcon and Ajay Yadava.
> 
> 
> Bugs: FALCON-1417
> https://issues.apache.org/jira/browse/FALCON-1417
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> Added optional date functionality.
> 
> 
> Diffs
> -
> 
>   
> common/src/main/java/org/apache/falcon/entity/parser/CrossEntityValidations.java
>  2696552 
>   common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 
> 992fc51 
>   
> common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java 
> 48a4286 
>   common/src/main/java/org/apache/falcon/util/DateUtil.java e736340 
>   
> common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java
>  d203b7c 
>   
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java
>  77f6a77 
> 
> Diff: https://reviews.apache.org/r/38459/diff/
> 
> 
> Testing
> ---
> 
> Added test cases for the same.
> 
> 
> Thanks,
> 
> PRAGYA MITTAL
> 
>

[jira] [Commented] (FALCON-1463) UI - unable to create feed with empty schema location

2015-09-17 Thread Ajay Yadava (JIRA) 

[ 
https://issues.apache.org/jira/browse/FALCON-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804941#comment-14804941
 ] 

Ajay Yadava commented on FALCON-1463:
-

[~venkatnrangan] Can you please take a look at it?

> UI - unable to create feed with empty schema location
> -
>
> Key: FALCON-1463
> URL: https://issues.apache.org/jira/browse/FALCON-1463
> Project: Falcon
>  Issue Type: Bug
>  Components: falcon-ui
>Affects Versions: 0.7
>Reporter: Michał Ziemba
>
> Using the new UI, I'm unable to create a feed that has empty location 
> attribute in schema tag:
> 
> (example of such feed is given here: 
> http://falcon.apache.org/HiveIntegration.html)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38441: Workflow Engine API is not supposed to expose the internal details

2015-09-17 Thread Pallavi Rao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38441/#review99483
---

Ship it!


Ship It!

- Pallavi Rao


On Sept. 16, 2015, 10:12 p.m., Balu Vellanki wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38441/
> ---
> 
> (Updated Sept. 16, 2015, 10:12 p.m.)
> 
> 
> Review request for Falcon and Pallavi Rao.
> 
> 
> Bugs: FALCON-1442
> https://issues.apache.org/jira/browse/FALCON-1442
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> The bug is related to patch for FALCON-1371. The patch introduces 4-5 methods 
> such as the one below:
> public abstract boolean isActive(Map bundles) throws 
> FalconException;
> 
> This totally breaks the contract of the WorkflowEngine. The Workflow Engine 
> API is not supposed to expose the internal details (whether the entity is 
> submitted as bundle or not). The worflow engine API should accept only 
> entities known to Falcon (such as cluster, entity, lifecycle).
> The new methods cause problem with the native scheduler (which extends 
> AbsractWorkflowEngine) as the native scheduler does not work with bundles.
> 
> 
> Diffs
> -
> 
>   common/pom.xml 0420b4c 
>   
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
>  ea86c2a 
>   
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java
>  5f79ca1 
>   prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 
> b867055 
> 
> Diff: https://reviews.apache.org/r/38441/diff/
> 
> 
> Testing
> ---
> 
> This patch does the following
> - removes reference to bundleJob from AbstractWorkflowEngine
> - Entity status are now limited to SUBMITTED, SUSPENDED, RUNNING, COMPLETED 
> as suggested by Srikanth Sundarrajan
> - removed unused dependency on oozie-client from falcon/common/pom.xml
> - findLatestBundle method is now private.
> 
> No new unit tests are required. Tested end2end after building the code.
> 
> 
> Thanks,
> 
> Balu Vellanki
> 
>

[jira] [Commented] (FALCON-1463) UI - unable to create feed with empty schema location

2015-09-17 Thread Venkat Ranganathan (JIRA) 

[ 
https://issues.apache.org/jira/browse/FALCON-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804978#comment-14804978
 ] 

Venkat Ranganathan commented on FALCON-1463:


This is fixed in the new UI that we have for patch review.   Let me given an 
overview of the UI and we can proceed to get it reviewed and committed so that 
it is generally available.

> UI - unable to create feed with empty schema location
> -
>
> Key: FALCON-1463
> URL: https://issues.apache.org/jira/browse/FALCON-1463
> Project: Falcon
>  Issue Type: Bug
>  Components: falcon-ui
>Affects Versions: 0.7
>Reporter: Michał Ziemba
>Assignee: Balu Vellanki
>
> Using the new UI, I'm unable to create a feed that has empty location 
> attribute in schema tag:
> 
> (example of such feed is given here: 
> http://falcon.apache.org/HiveIntegration.html)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)