date:20150914


[ 
https://issues.apache.org/jira/browse/FALCON-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743343#comment-14743343
 ] 

Pallavi Rao commented on FALCON-1401:
-

{code}
if (entity.getInputs() == null) {
props.put(WorkflowExecutionArgs.INPUT_FEED_NAMES.getName(), "NONE");
props.put(WorkflowExecutionArgs.INPUT_FEED_PATHS.getName(), IGNORE);
props.put(WorkflowExecutionArgs.INPUT_NAMES.getName(), IGNORE);
return;
}

if (entity.getOutputs() == null) {
props.put(WorkflowExecutionArgs.OUTPUT_FEED_NAMES.getName(), 
IGNORE);
props.put(WorkflowExecutionArgs.OUTPUT_FEED_PATHS.getName(), 
IGNORE);
return;
}
{code}

Input feed paths and names have always been set to IGNORE. Looks like the one 
violating the consistency now is :
{code}
props.put(WorkflowExecutionArgs.INPUT_FEED_NAMES.getName(), "NONE");
{code}

If we can change this to IGNORE too and have the MetadataMappingService check 
for IGNORE rather than NONE, it will all be consistent.

> MetadataMappingService fails to add an edge for a process instance
> --
>
> Key: FALCON-1401
> URL: https://issues.apache.org/jira/browse/FALCON-1401
> Project: Falcon
>  Issue Type: Bug
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
>Priority: Minor
> Attachments: FALCON-1401.patch
>
>
> If a process has no output feeds, the metadata mapping service fails to add 
> an edge for the process instances. It fails with NPE:
> {code}
> 2015-08-14 15:31:00,950 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:00,951 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,957 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:00,981 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,984 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,988 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,990 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,992 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,995 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,997 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,009 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:01,009 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,012 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:01,033 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,035 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,039 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,041 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,043 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,044 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,047 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
>

Re: Review Request 38105: FALCON-1425: Provide Email based notification plugin to send notification when Falcon instance completes.

2015-09-14 Thread Ajay Yadava

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38105/#review98677
---

This is a very useful feature. It will be very useful to have a one page 
documentation in the docs on how to use it and it's behavior (e.g. email coming 
only after all retries or otherwise)

client/src/main/java/org/apache/falcon/entity/v0/EntityNotification.java (line 
23)

nit: this documentation can use some more explanation.
What is meant by notification list - "recepients list" or "type of 
notifications"?

client/src/main/java/org/apache/falcon/entity/v0/EntityNotification.java (line 
26)

What is type? What is it used for?

client/src/main/java/org/apache/falcon/entity/v0/EntityNotification.java (line 
29)

It will be more useful to have marshalled xml in toString, since it is a 
type for xsd and is used in xmls.

client/src/main/resources/feed-0.1.xsd (line 303)

type should be of xs:enumeration instead of xs:string because any random 
string won't be accepted by falcon.

client/src/main/resources/feed-0.1.xsd (line 304)

It will be good to document that the to list can contain multiple email 
addresseses separated by comma.

client/src/main/resources/process-0.1.xsd (line 412)

same as feed.xsd

metrics/src/main/java/org/apache/falcon/plugin/NotificationPlugin.java (line 25)

There is no "Notification concrete class". Perhaps you meant 
EmailNotification.java. 

In any case it is not a good documentation for this interface. A short 
description of what is the contract that this interface provides will be useful.

metrics/src/main/java/org/apache/falcon/plugin/NotificationPlugin.java (line 27)

This interface has no characterstics of a plugin unlike others e.g. 
MonitoringPlugin. It should be named as just Notification.

metrics/src/main/java/org/apache/falcon/util/EmailNotificationArgs.java (line 
24)

These are not arguments for Email Notification, these are just constants 
for SMTP properties. Should be renamed accordingly.

oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
 (line 286)

this test should be part of FeedEntityParserTest. It has nothing to do with 
workflowbuilding or oozie.

oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java
 (line 222)

Should be part of ProcessEntityParserTest.

prism/src/main/java/org/apache/falcon/plugin/EmailNotification.java (line 45)

There are 2 classes:

1) EmailNotification implements NotificationPlugin
2) EmailNotificationPlugin implements MonitoringPlugin

They are both confusing in terms of name. There should be just one class 
which implements both monitoring plugin and NotificationPlugin and called 
EmailNotificationPlugin.

prism/src/main/java/org/apache/falcon/plugin/EmailNotification.java (line 110)

This should be under the "if - else if" to avoid sending email for alerts 
other than "wf-instance-succeeded" and "wf-instance-failed"

prism/src/main/java/org/apache/falcon/util/NotificationUtil.java (line 44)

Just an observation, this pattern will allow some invalid domains. 

e.g. ajay.ya...@gmail.com.1

Although a perfect validation is not required as this is still a runtime 
check and mail will fail in any case. Following might be a better regex.

"^[_A-Za-z0-9-\+]+(\.[_A-Za-z0-9-]+)*@"
+ "[A-Za-z0-9-]+(\.[A-Za-z0-9]+)*(\.[A-Za-z]{2,})$"

- Ajay Yadava

On Sept. 11, 2015, 3:09 p.m., Peeyush Bishnoi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38105/
> ---
> 
> (Updated Sept. 11, 2015, 3:09 p.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1425
> https://issues.apache.org/jira/browse/FALCON-1425
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> Provide Email based notification plugin to send notification when Falcon 
> instance completes.
> 
> 
> Diffs
> -
> 
>

[jira] [Commented] (FALCON-1425) Provide Email based plugin to send Notification once instance completed


[ 
https://issues.apache.org/jira/browse/FALCON-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743532#comment-14743532
 ] 

Ajay Yadava commented on FALCON-1425:
-

[~peeyushb] This is very useful and can be leveraged for enabling other 
features as well. Patch looks ok apart from some nits posted on review board. 

> Provide Email based plugin to send Notification once instance completed
> ---
>
> Key: FALCON-1425
> URL: https://issues.apache.org/jira/browse/FALCON-1425
> Project: Falcon
>  Issue Type: Sub-task
>  Components: common
>Reporter: Peeyush Bishnoi
>Assignee: Peeyush Bishnoi
> Fix For: trunk
>
> Attachments: FALCON-1425.patch, FALCON-1425.v1.patch, 
> FALCON-1425.v2.patch
>
>
> Requirement is to provide Email based notification plugin to send 
> notification when Falcon instance completes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38294: FALCON-1434 Enhance schedule API to accept key-value properties

2015-09-14 Thread Balu Vellanki


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38294/#review98854
---

Ship it!


Ship It!

- Balu Vellanki


On Sept. 14, 2015, 6:21 a.m., Pallavi Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38294/
> ---
> 
> (Updated Sept. 14, 2015, 6:21 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1434
> https://issues.apache.org/jira/browse/FALCON-1434
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> The schedule API will be enhanced to accept a key-value properties. This is a 
> foundation to enable users to specify the scheduler on which they want to 
> schedule the entity. This in turn enables migration to native scheduler from 
> Oozie.
> Example:
> bin/falcon entity -schedule -props falcon.scheduler=native -name 
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/falcon/cli/FalconCLI.java d4da302 
>   client/src/main/java/org/apache/falcon/client/AbstractFalconClient.java 
> 282b41b 
>   client/src/main/java/org/apache/falcon/client/FalconClient.java 44436d2 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 25d9008 
>   
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
>  ea86c2a 
>   common/src/test/java/org/apache/falcon/entity/EntityUtilTest.java cfdc84d 
>   
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java
>  5f79ca1 
>   
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java
>  f9405dc 
>   
> prism/src/main/java/org/apache/falcon/resource/proxy/SchedulableEntityManagerProxy.java
>  ceabb06 
>   unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java eb65cb3 
>   unit/src/test/java/org/apache/falcon/unit/FalconUnitTestBase.java 997b301 
>   unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java 498f50e 
>   
> webapp/src/main/java/org/apache/falcon/resource/SchedulableEntityManager.java 
> 1f8cc1b 
>   webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 0062070 
>   webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java 
> bcd3bd5 
>   webapp/src/test/java/org/apache/falcon/resource/TestContext.java 54671fb 
> 
> Diff: https://reviews.apache.org/r/38294/diff/
> 
> 
> Testing
> ---
> 
> UT added
> Manually tested to ensure CLI accepts properties and it is propagated.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>

[jira] [Updated] (FALCON-1343) validation of read/write endpoints is not reliable


 [ 
https://issues.apache.org/jira/browse/FALCON-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balu Vellanki updated FALCON-1343:
--
Attachment: FALCON-1343.patch

> validation of read/write endpoints is not reliable
> --
>
> Key: FALCON-1343
> URL: https://issues.apache.org/jira/browse/FALCON-1343
> Project: Falcon
>  Issue Type: Sub-task
>  Components: general
>Reporter: Balu Vellanki
>Assignee: Balu Vellanki
> Attachments: FALCON-1343.patch
>
>
> A read/write endpoint is currently validated by creating a filesystem with 
> the endpoint url. 
> {code}
> HadoopClientFactory.get().createProxiedFileSystem(conf);
> {code}
> I confirmed with a HDFS team member that it is not sufficient validation.  
> Ideally check if the end user has access by doing atleast a list /tmp call 
> after creating proxiedFileSystem. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1343) validation of read/write endpoints is not reliable


 [ 
https://issues.apache.org/jira/browse/FALCON-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balu Vellanki updated FALCON-1343:
--
Summary: validation of read/write endpoints is not reliable  (was: 
validation of read/write endpoints is not correct)

> validation of read/write endpoints is not reliable
> --
>
> Key: FALCON-1343
> URL: https://issues.apache.org/jira/browse/FALCON-1343
> Project: Falcon
>  Issue Type: Sub-task
>  Components: general
>Reporter: Balu Vellanki
>Assignee: Balu Vellanki
>
> A read/write endpoint is currently validated by creating a filesystem with 
> the endpoint url. 
> {code}
> HadoopClientFactory.get().createProxiedFileSystem(conf);
> {code}
> I confirmed with a HDFS team member that it is not sufficient validation.  
> Ideally check if the end user has access by doing atleast a list /tmp call 
> after creating proxiedFileSystem. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FALCON-1448) OozieWorkflowEngine Refactoring

Ajay Yadava created FALCON-1448:
---

 Summary: OozieWorkflowEngine Refactoring
 Key: FALCON-1448
 URL: https://issues.apache.org/jira/browse/FALCON-1448
 Project: Falcon
  Issue Type: Improvement
Reporter: Ajay Yadava






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38387: validation of read/write endpoints is not reliable - fix this

2015-09-14 Thread Balu Vellanki


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38387/
---

(Updated Sept. 15, 2015, 12:09 a.m.)


Review request for Falcon, Ajay Yadava and Sowmya Ramesh.


Bugs: FALCON-1343
https://issues.apache.org/jira/browse/FALCON-1343


Repository: falcon-git


Description
---

A read/write endpoint is currently validated by creating a filesystem with the 
endpoint url. 
{code}
HadoopClientFactory.get().createProxiedFileSystem(conf);
{code}

I confirmed with a HDFS team member that it is not sufficient validation.  
Ideally check if the end user has access by doing atleast a list /tmp call 
after creating proxiedFileSystem.


Diffs (updated)
-

  common/src/main/java/org/apache/falcon/entity/parser/ClusterEntityParser.java 
5756f84 
  common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java a36623c 
  
common/src/test/java/org/apache/falcon/entity/parser/ClusterEntityParserTest.java
 638cef9 
  common/src/test/resources/config/cluster/cluster-bad-write-endpoint.xml 
PRE-CREATION 

Diff: https://reviews.apache.org/r/38387/diff/


Testing
---

Tested end2end, added a unit test.


Thanks,

Balu Vellanki

[jira] [Updated] (FALCON-1343) validation of read/write endpoints is not reliable


 [ 
https://issues.apache.org/jira/browse/FALCON-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balu Vellanki updated FALCON-1343:
--
Attachment: (was: FALCON-1343.patch)

> validation of read/write endpoints is not reliable
> --
>
> Key: FALCON-1343
> URL: https://issues.apache.org/jira/browse/FALCON-1343
> Project: Falcon
>  Issue Type: Sub-task
>  Components: general
>Reporter: Balu Vellanki
>Assignee: Balu Vellanki
> Attachments: FALCON-1343.patch
>
>
> A read/write endpoint is currently validated by creating a filesystem with 
> the endpoint url. 
> {code}
> HadoopClientFactory.get().createProxiedFileSystem(conf);
> {code}
> I confirmed with a HDFS team member that it is not sufficient validation.  
> Ideally check if the end user has access by doing atleast a list /tmp call 
> after creating proxiedFileSystem. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FALCON-1449) Move getEntityProperties method to EntityUtil

Ajay Yadava created FALCON-1449:
---

 Summary: Move getEntityProperties method to EntityUtil
 Key: FALCON-1449
 URL: https://issues.apache.org/jira/browse/FALCON-1449
 Project: Falcon
  Issue Type: Sub-task
Reporter: Ajay Yadava
Assignee: Ajay Yadava


getProperties method is currently in OozieEntityBuilder however it has nothing 
specific to oozie adaptor for falcon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 38387: validation of read/write endpoints is not reliable - fix this

2015-09-14 Thread Balu Vellanki


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38387/
---

Review request for Falcon, Ajay Yadava and Sowmya Ramesh.


Bugs: FALCON-1343
https://issues.apache.org/jira/browse/FALCON-1343


Repository: falcon-git


Description
---

A read/write endpoint is currently validated by creating a filesystem with the 
endpoint url. 
{code}
HadoopClientFactory.get().createProxiedFileSystem(conf);
{code}

I confirmed with a HDFS team member that it is not sufficient validation.  
Ideally check if the end user has access by doing atleast a list /tmp call 
after creating proxiedFileSystem.


Diffs
-

  common/src/main/java/org/apache/falcon/entity/parser/ClusterEntityParser.java 
5756f84 
  common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java a36623c 
  
common/src/test/java/org/apache/falcon/entity/parser/ClusterEntityParserTest.java
 638cef9 
  common/src/test/resources/config/cluster/cluster-bad-write-endpoint.xml 
1d15e16 

Diff: https://reviews.apache.org/r/38387/diff/


Testing
---

Tested end2end, added a unit test.


Thanks,

Balu Vellanki

[jira] [Commented] (FALCON-1440) Better json for triage api


[ 
https://issues.apache.org/jira/browse/FALCON-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744301#comment-14744301
 ] 

Balu Vellanki commented on FALCON-1440:
---

[~ajayyadava] : I did not fully understand your comment. Can you please explain 
a bit more?

 In the response, each one of "name", "type", "cluster" etc is a vertex that is 
associated with the given feed. But looking at the response, once can be led to 
thinking these are characteristics of a vertex and there is only one vertex in 
the array. Am I missing something here?

> Better json for triage api
> --
>
> Key: FALCON-1440
> URL: https://issues.apache.org/jira/browse/FALCON-1440
> Project: Falcon
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Ajay Yadava
> Fix For: 0.7
>
>
> Firing rest request for triage returns a response which looks like:
> {code}
>   "vertices": [
> "name: A78e9f5a1-5b8eab89, type: FEED, cluster: A78e9f5a1-9238cc6e, 
> instanceTime: 2010-01-02T00:40Z, tags: [AVAILABLE]"
>   ]
> {code}
> The cli output can also be formatted better:
> {code}
> digraph g{ 
> "name: A78e9f5a1-83173830, type: FEED, cluster: A78e9f5a1-9238cc6e, 
> instanceTime: 2010-01-02T00:40Z, tags: [MISSING]"
> }
> {code}
> Here is the full request/response that was made:
> instance/triage/feed/A78e9f5a1-5b8eab89?start=2010-01-02T00%3A40Z=2015-01-05T01%3A00Z=hrt_qa
> {code}
> {
>   "triageGraphs": [
> {
>   "vertices": [
> "name: A78e9f5a1-5b8eab89, type: FEED, cluster: A78e9f5a1-9238cc6e, 
> instanceTime: 2010-01-02T00:40Z, tags: [AVAILABLE]"
>   ]
> }
>   ],
>   "requestId": "default/1712257401@qtp-1961945640-622 - 
> 4b19ad04-2433-41c8-ad11-541fc0509e0d\n",
>   "message": "default/Success\n",
>   "status": "SUCCEEDED"
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1448) OozieWorkflowEngine Refactoring


 [ 
https://issues.apache.org/jira/browse/FALCON-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Yadava updated FALCON-1448:

Description: Placeholder umbrella JIRA for various refactoring in 
OozieWorkflowEngine. Will add sub-tasks for each of the items.  (was: Umbrella 
JIRA for various refactoring in OozieWorkflowEngine. Will add sub-tasks for 
each of the items.)

> OozieWorkflowEngine Refactoring
> ---
>
> Key: FALCON-1448
> URL: https://issues.apache.org/jira/browse/FALCON-1448
> Project: Falcon
>  Issue Type: Improvement
>Reporter: Ajay Yadava
>
> Placeholder umbrella JIRA for various refactoring in OozieWorkflowEngine. 
> Will add sub-tasks for each of the items.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1448) OozieWorkflowEngine Refactoring


 [ 
https://issues.apache.org/jira/browse/FALCON-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Yadava updated FALCON-1448:

Description: Umbrella JIRA for various refactoring in OozieWorkflowEngine. 
Will add sub-tasks for each of the items.

> OozieWorkflowEngine Refactoring
> ---
>
> Key: FALCON-1448
> URL: https://issues.apache.org/jira/browse/FALCON-1448
> Project: Falcon
>  Issue Type: Improvement
>Reporter: Ajay Yadava
>
> Umbrella JIRA for various refactoring in OozieWorkflowEngine. Will add 
> sub-tasks for each of the items.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1343) validation of read/write endpoints is not reliable


 [ 
https://issues.apache.org/jira/browse/FALCON-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balu Vellanki updated FALCON-1343:
--
Attachment: FALCON-1343.patch

> validation of read/write endpoints is not reliable
> --
>
> Key: FALCON-1343
> URL: https://issues.apache.org/jira/browse/FALCON-1343
> Project: Falcon
>  Issue Type: Sub-task
>  Components: general
>Reporter: Balu Vellanki
>Assignee: Balu Vellanki
> Attachments: FALCON-1343.patch
>
>
> A read/write endpoint is currently validated by creating a filesystem with 
> the endpoint url. 
> {code}
> HadoopClientFactory.get().createProxiedFileSystem(conf);
> {code}
> I confirmed with a HDFS team member that it is not sufficient validation.  
> Ideally check if the end user has access by doing atleast a list /tmp call 
> after creating proxiedFileSystem. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1448) OozieWorkflowEngine Refactoring


[ 
https://issues.apache.org/jira/browse/FALCON-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744815#comment-14744815
 ] 

Pallavi Rao commented on FALCON-1448:
-

[~ajayyadava], is there any feature or use case that is driving this refactor. 
Or, is it just code cleanup? Is the refactoring going to touch the API too? 

> OozieWorkflowEngine Refactoring
> ---
>
> Key: FALCON-1448
> URL: https://issues.apache.org/jira/browse/FALCON-1448
> Project: Falcon
>  Issue Type: Improvement
>Reporter: Ajay Yadava
>
> Placeholder umbrella JIRA for various refactoring in OozieWorkflowEngine. 
> Will add sub-tasks for each of the items.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FALCON-1458) Update documentation on site and announce the 0.8 release

Sowmya Ramesh created FALCON-1458:
-

 Summary: Update documentation on site and announce the 0.8 release
 Key: FALCON-1458
 URL: https://issues.apache.org/jira/browse/FALCON-1458
 Project: Falcon
  Issue Type: Sub-task
  Components: ease
Reporter: Sowmya Ramesh
Assignee: Sowmya Ramesh


* Update the documentation on site. 
* Announce the release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Discuss] : Should a non-superuser be allowed to update ACL of feed or process entity

Agree that "user2" shouldn't be allowed to just update the entity and
change the ownership. All the more reason to have a separate Auth API,
rather than embed the ACL in the entity itself. Such issues can be handled
in a much cleaner way.

Regards,
Pallavi

On Tue, Sep 15, 2015 at 3:12 AM, Balu Vellanki 
wrote:

> Hi Team,
>
> Today, Feed/Process entities have ACL with owner and group. Support for
> permissions is not implemented yet. Any user who is the owner OR who
> belongs to the group can update/delete/suspend the entity.
>
> If two users "user1" and "user2" belong to same group "users" and the
> falcon entity ACL is ,
> then user2 can update the falcon entity and claim ownership of this entity.
> I believe that user2 should not be allowed to do so unless it is
> superuser.  Similar behavior is not allowed in HDFS.  Please comment if you
> disagree.
>
> https://issues.apache.org/jira/browse/FALCON-1340
>
> Thanks
> Balu Velalnki
>

-- 
_
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.

[jira] [Updated] (FALCON-1027) Falcon REST API trusted proxy support


 [ 
https://issues.apache.org/jira/browse/FALCON-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sowmya Ramesh updated FALCON-1027:
--
Attachment: (was: FALCON-1027.v3.patch)

> Falcon REST API trusted proxy support
> -
>
> Key: FALCON-1027
> URL: https://issues.apache.org/jira/browse/FALCON-1027
> Project: Falcon
>  Issue Type: New Feature
>Affects Versions: 0.7
>Reporter: kenneth ho
>Assignee: Sowmya Ramesh
> Fix For: trunk
>
> Attachments: ApacheFalcon-Proxyusersupport.pdf, FALCON-1027.v0.patch, 
> FALCON-1027.v1.patch, FALCON-1027.v2.patch
>
>
> In order for Falcon REST API to work securely via the Knox gateway it must be 
> possible to setup a trust relationship between Knox and Falcon. This is 
> commonly done in other Hadoop ecosystem components using a combination of 
> Kerberos/SPNego and a doas URL query parameter. This provides a mechanism for 
> Falcon to strongly authenticate Knox as a trusted proxy, ensuring that it can 
> trust the identity assertions made via the doas query parameter. The links 
> below provide some information describing how this is done for core Hadoop. 
> Also note that most components utilize Hadoop core's reusable hadoop-auth 
> module to implement this functionality.
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Proxy_Users
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Proxy_user



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FALCON-1454) Verify source tarball and run few end to end tests

Sowmya Ramesh created FALCON-1454:
-

 Summary: Verify source tarball and run few end to end tests
 Key: FALCON-1454
 URL: https://issues.apache.org/jira/browse/FALCON-1454
 Project: Falcon
  Issue Type: Sub-task
  Components: build-tools
Affects Versions: 0.7
Reporter: Sowmya Ramesh
Assignee: Sowmya Ramesh


Verify released source tarball and run few tests on distributed and standalone 
mode.
Additionally track +1 from community.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1418) Update Rest API twiki for proxy support


 [ 
https://issues.apache.org/jira/browse/FALCON-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sowmya Ramesh updated FALCON-1418:
--
Fix Version/s: trunk

> Update Rest API twiki for proxy support
> ---
>
> Key: FALCON-1418
> URL: https://issues.apache.org/jira/browse/FALCON-1418
> Project: Falcon
>  Issue Type: Sub-task
>Reporter: Sowmya Ramesh
>Assignee: Sowmya Ramesh
> Fix For: trunk
>
> Attachments: FALCON-1418.v0.patch, FALCON-1418.v1.patch
>
>
> Update the Rest API twiki with doAs optional query parameter usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38294: FALCON-1434 Enhance schedule API to accept key-value properties



> On Sept. 11, 2015, 6:12 p.m., pavan kumar kolamuri wrote:
> > client/src/main/java/org/apache/falcon/cli/FalconCLI.java, line 439
> > 
> >
> > Instead of taking props as String, won't it be good to take props from 
> > a file ? Like for instance rerun we are taking props from file.

These are properties meant for scheduler. Don't expect it to be too many. In 
fact, right now, I just have one in mind :-). If this list grows, then we can 
consider supporting a properties file.


- Pallavi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38294/#review98652
---


On Sept. 14, 2015, 6:21 a.m., Pallavi Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38294/
> ---
> 
> (Updated Sept. 14, 2015, 6:21 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1434
> https://issues.apache.org/jira/browse/FALCON-1434
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> The schedule API will be enhanced to accept a key-value properties. This is a 
> foundation to enable users to specify the scheduler on which they want to 
> schedule the entity. This in turn enables migration to native scheduler from 
> Oozie.
> Example:
> bin/falcon entity -schedule -props falcon.scheduler=native -name 
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/falcon/cli/FalconCLI.java d4da302 
>   client/src/main/java/org/apache/falcon/client/AbstractFalconClient.java 
> 282b41b 
>   client/src/main/java/org/apache/falcon/client/FalconClient.java 44436d2 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 25d9008 
>   
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
>  ea86c2a 
>   common/src/test/java/org/apache/falcon/entity/EntityUtilTest.java cfdc84d 
>   
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java
>  5f79ca1 
>   
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java
>  f9405dc 
>   
> prism/src/main/java/org/apache/falcon/resource/proxy/SchedulableEntityManagerProxy.java
>  ceabb06 
>   unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java eb65cb3 
>   unit/src/test/java/org/apache/falcon/unit/FalconUnitTestBase.java 997b301 
>   unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java 498f50e 
>   
> webapp/src/main/java/org/apache/falcon/resource/SchedulableEntityManager.java 
> 1f8cc1b 
>   webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 0062070 
>   webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java 
> bcd3bd5 
>   webapp/src/test/java/org/apache/falcon/resource/TestContext.java 54671fb 
> 
> Diff: https://reviews.apache.org/r/38294/diff/
> 
> 
> Testing
> ---
> 
> UT added
> Manually tested to ensure CLI accepts properties and it is propagated.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>

[jira] [Created] (FALCON-1453) Upgrade falcon POM for 0.8 release

Sowmya Ramesh created FALCON-1453:
-

 Summary: Upgrade falcon POM for 0.8 release
 Key: FALCON-1453
 URL: https://issues.apache.org/jira/browse/FALCON-1453
 Project: Falcon
  Issue Type: Sub-task
  Components: build-tools
Reporter: Sowmya Ramesh
Assignee: Sowmya Ramesh


Upgrade and release of POM for Falcon 0.8




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FALCON-1452) Document upgrade from 0.7 to 0.8

Sowmya Ramesh created FALCON-1452:
-

 Summary: Document upgrade from 0.7 to 0.8
 Key: FALCON-1452
 URL: https://issues.apache.org/jira/browse/FALCON-1452
 Project: Falcon
  Issue Type: Sub-task
  Components: docs
Reporter: Sowmya Ramesh
Assignee: Sowmya Ramesh


Document the upgrade steps from 0.7 to 0.8 and also call out any backward 
incompatibility and workarounds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Discuss] : Should a non-superuser be allowed to update ACL of feed or process entity

2015-09-14 Thread pavan kumar Kolamuri

Even i agree that user2 shouldn't update/delete/suspend the entity, but we
should be consistent across all API's for the same. As of now submit is
allowed if user belongs to the same group of ACL owner group right ? Should
we also change this behaviour to make sure only ACL owner should be allowed
to submit ?

On Tue, Sep 15, 2015 at 9:58 AM, Pallavi Rao  wrote:

> Agree that "user2" shouldn't be allowed to just update the entity and
> change the ownership. All the more reason to have a separate Auth API,
> rather than embed the ACL in the entity itself. Such issues can be handled
> in a much cleaner way.
>
> Regards,
> Pallavi
>
> On Tue, Sep 15, 2015 at 3:12 AM, Balu Vellanki 
> wrote:
>
> > Hi Team,
> >
> > Today, Feed/Process entities have ACL with owner and group. Support for
> > permissions is not implemented yet. Any user who is the owner OR who
> > belongs to the group can update/delete/suspend the entity.
> >
> > If two users "user1" and "user2" belong to same group "users" and the
> > falcon entity ACL is ,
> > then user2 can update the falcon entity and claim ownership of this
> entity.
> > I believe that user2 should not be allowed to do so unless it is
> > superuser.  Similar behavior is not allowed in HDFS.  Please comment if
> you
> > disagree.
> >
> > https://issues.apache.org/jira/browse/FALCON-1340
> >
> > Thanks
> > Balu Velalnki
> >
>
> --
> _
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>



-- 
Regards
Pavan Kumar Kolamuri

[jira] [Updated] (FALCON-1418) Update Rest API twiki for proxy support


 [ 
https://issues.apache.org/jira/browse/FALCON-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sowmya Ramesh updated FALCON-1418:
--
Attachment: FALCON-1418.v1.patch

> Update Rest API twiki for proxy support
> ---
>
> Key: FALCON-1418
> URL: https://issues.apache.org/jira/browse/FALCON-1418
> Project: Falcon
>  Issue Type: Sub-task
>Reporter: Sowmya Ramesh
>Assignee: Sowmya Ramesh
> Attachments: FALCON-1418.v0.patch, FALCON-1418.v1.patch
>
>
> Update the Rest API twiki with doAs optional query parameter usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FALCON-1450) Prepare Falcon Release v0.8

Sowmya Ramesh created FALCON-1450:
-

 Summary: Prepare Falcon Release v0.8
 Key: FALCON-1450
 URL: https://issues.apache.org/jira/browse/FALCON-1450
 Project: Falcon
  Issue Type: Task
  Components: ease
Affects Versions: 0.7
Reporter: Sowmya Ramesh
Assignee: Sowmya Ramesh


This is a parent holding ticket for tracking all tasks relating to 0.8 release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FALCON-1457) Update changes.txt for 0.8 branch

Sowmya Ramesh created FALCON-1457:
-

 Summary: Update changes.txt for 0.8 branch
 Key: FALCON-1457
 URL: https://issues.apache.org/jira/browse/FALCON-1457
 Project: Falcon
  Issue Type: Sub-task
  Components: ease
Reporter: Sowmya Ramesh
Assignee: Sowmya Ramesh


Update changes.txt for 0.8 branch



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FALCON-1455) Create branch 0.8

Sowmya Ramesh created FALCON-1455:
-

 Summary: Create branch 0.8
 Key: FALCON-1455
 URL: https://issues.apache.org/jira/browse/FALCON-1455
 Project: Falcon
  Issue Type: Sub-task
  Components: general
Reporter: Sowmya Ramesh
Assignee: Sowmya Ramesh


Create branch 0.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1027) Falcon REST API trusted proxy support


[ 
https://issues.apache.org/jira/browse/FALCON-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744674#comment-14744674
 ] 

Sowmya Ramesh commented on FALCON-1027:
---

commit d8fbec9f97ba2bd2207827c7b83a8b861c25f9f0

> Falcon REST API trusted proxy support
> -
>
> Key: FALCON-1027
> URL: https://issues.apache.org/jira/browse/FALCON-1027
> Project: Falcon
>  Issue Type: New Feature
>Affects Versions: 0.7
>Reporter: kenneth ho
>Assignee: Sowmya Ramesh
> Fix For: trunk
>
> Attachments: ApacheFalcon-Proxyusersupport.pdf, FALCON-1027.V3.patch, 
> FALCON-1027.v0.patch, FALCON-1027.v1.patch, FALCON-1027.v2.patch
>
>
> In order for Falcon REST API to work securely via the Knox gateway it must be 
> possible to setup a trust relationship between Knox and Falcon. This is 
> commonly done in other Hadoop ecosystem components using a combination of 
> Kerberos/SPNego and a doas URL query parameter. This provides a mechanism for 
> Falcon to strongly authenticate Knox as a trusted proxy, ensuring that it can 
> trust the identity assertions made via the doas query parameter. The links 
> below provide some information describing how this is done for core Hadoop. 
> Also note that most components utilize Hadoop core's reusable hadoop-auth 
> module to implement this functionality.
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Proxy_Users
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Proxy_user



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1418) Update Rest API twiki for proxy support


[ 
https://issues.apache.org/jira/browse/FALCON-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744675#comment-14744675
 ] 

Sowmya Ramesh commented on FALCON-1418:
---

commit d8fbec9f97ba2bd2207827c7b83a8b861c25f9f0. Resolving this as Parent is 
resolved.

> Update Rest API twiki for proxy support
> ---
>
> Key: FALCON-1418
> URL: https://issues.apache.org/jira/browse/FALCON-1418
> Project: Falcon
>  Issue Type: Sub-task
>Reporter: Sowmya Ramesh
>Assignee: Sowmya Ramesh
> Attachments: FALCON-1418.v0.patch, FALCON-1418.v1.patch
>
>
> Update the Rest API twiki with doAs optional query parameter usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1342) Do not allow duplicate properties in cluster entities


[ 
https://issues.apache.org/jira/browse/FALCON-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744783#comment-14744783
 ] 

Ajay Yadava commented on FALCON-1342:
-

[~bvellanki] This change seems to address the duplicate property issue only in 
cluster entity. Any reason for not doing it for process and feed also?

> Do not allow duplicate properties in cluster entities
> -
>
> Key: FALCON-1342
> URL: https://issues.apache.org/jira/browse/FALCON-1342
> Project: Falcon
>  Issue Type: Sub-task
>  Components: general
>Reporter: Balu Vellanki
>Assignee: Balu Vellanki
> Attachments: FALCON-1342.patch
>
>
> When specifying properties for a cluster, this is currently allowed,
> {code}
> 
> 
> 
> 
> {code}
> The propeties are stored as an array of 
> org.apache.falcon.entity.v0.cluster.Property, and cluster.getProperty("test") 
> will return either "value1" or "value2" but not both. If falcon does not 
> support multiple values for same property key,  parsing such an entity should 
> throw an error. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1027) Falcon REST API trusted proxy support


 [ 
https://issues.apache.org/jira/browse/FALCON-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sowmya Ramesh updated FALCON-1027:
--
Attachment: FALCON-1027.V3.patch

> Falcon REST API trusted proxy support
> -
>
> Key: FALCON-1027
> URL: https://issues.apache.org/jira/browse/FALCON-1027
> Project: Falcon
>  Issue Type: New Feature
>Affects Versions: 0.7
>Reporter: kenneth ho
>Assignee: Sowmya Ramesh
> Fix For: trunk
>
> Attachments: ApacheFalcon-Proxyusersupport.pdf, FALCON-1027.V3.patch, 
> FALCON-1027.v0.patch, FALCON-1027.v1.patch, FALCON-1027.v2.patch
>
>
> In order for Falcon REST API to work securely via the Knox gateway it must be 
> possible to setup a trust relationship between Knox and Falcon. This is 
> commonly done in other Hadoop ecosystem components using a combination of 
> Kerberos/SPNego and a doas URL query parameter. This provides a mechanism for 
> Falcon to strongly authenticate Knox as a trusted proxy, ensuring that it can 
> trust the identity assertions made via the doas query parameter. The links 
> below provide some information describing how this is done for core Hadoop. 
> Also note that most components utilize Hadoop core's reusable hadoop-auth 
> module to implement this functionality.
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Proxy_Users
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Proxy_user



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Discuss] : Should a non-superuser be allowed to update ACL of feed or process entity

2015-09-14 Thread Sandeep Samudrala

I agree with above point to handle submission time. But again an entity can
be submitted and scheduled with different users, in which case the user
with which schedule is ran will be used. We might have to handle even
scheduling part. I think rather than handling ACL at various levels, the
whole ACL can be improved as part of FALCON-1367
.

On Tue, Sep 15, 2015 at 10:14 AM, pavan kumar Kolamuri <
pavan.kolam...@gmail.com> wrote:

> Even i agree that user2 shouldn't update/delete/suspend the entity, but we
> should be consistent across all API's for the same. As of now submit is
> allowed if user belongs to the same group of ACL owner group right ? Should
> we also change this behaviour to make sure only ACL owner should be allowed
> to submit ?
>
> On Tue, Sep 15, 2015 at 9:58 AM, Pallavi Rao 
> wrote:
>
> > Agree that "user2" shouldn't be allowed to just update the entity and
> > change the ownership. All the more reason to have a separate Auth API,
> > rather than embed the ACL in the entity itself. Such issues can be
> handled
> > in a much cleaner way.
> >
> > Regards,
> > Pallavi
> >
> > On Tue, Sep 15, 2015 at 3:12 AM, Balu Vellanki <
> bvella...@hortonworks.com>
> > wrote:
> >
> > > Hi Team,
> > >
> > > Today, Feed/Process entities have ACL with owner and group. Support for
> > > permissions is not implemented yet. Any user who is the owner OR who
> > > belongs to the group can update/delete/suspend the entity.
> > >
> > > If two users "user1" and "user2" belong to same group "users" and the
> > > falcon entity ACL is ,
> > > then user2 can update the falcon entity and claim ownership of this
> > entity.
> > > I believe that user2 should not be allowed to do so unless it is
> > > superuser.  Similar behavior is not allowed in HDFS.  Please comment if
> > you
> > > disagree.
> > >
> > > https://issues.apache.org/jira/browse/FALCON-1340
> > >
> > > Thanks
> > > Balu Velalnki
> > >
> >
> > --
> > _
> > The information contained in this communication is intended solely for
> the
> > use of the individual or entity to whom it is addressed and others
> > authorized to receive it. It may contain confidential or legally
> privileged
> > information. If you are not the intended recipient you are hereby
> notified
> > that any disclosure, copying, distribution or taking any action in
> reliance
> > on the contents of this information is strictly prohibited and may be
> > unlawful. If you have received this communication in error, please notify
> > us immediately by responding to this email and then delete it from your
> > system. The firm is neither liable for the proper and complete
> transmission
> > of the information contained in this communication nor for any delay in
> its
> > receipt.
> >
>
>
>
> --
> Regards
> Pavan Kumar Kolamuri
>

[jira] [Created] (FALCON-1451) Update JIRA fix versions

Sowmya Ramesh created FALCON-1451:
-

 Summary: Update JIRA fix versions
 Key: FALCON-1451
 URL: https://issues.apache.org/jira/browse/FALCON-1451
 Project: Falcon
  Issue Type: Sub-task
  Components: general
Affects Versions: 0.7
Reporter: Sowmya Ramesh
Assignee: Sowmya Ramesh


Check and update Fix Versions of all the JIRAs




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1342) Do not allow duplicate properties in cluster entities


 [ 
https://issues.apache.org/jira/browse/FALCON-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balu Vellanki updated FALCON-1342:
--
Attachment: FALCON-1342.patch

> Do not allow duplicate properties in cluster entities
> -
>
> Key: FALCON-1342
> URL: https://issues.apache.org/jira/browse/FALCON-1342
> Project: Falcon
>  Issue Type: Sub-task
>  Components: general
>Reporter: Balu Vellanki
>Assignee: Balu Vellanki
> Attachments: FALCON-1342.patch
>
>
> When specifying properties for a cluster, this is currently allowed,
> {code}
> 
> 
> 
> 
> {code}
> The propeties are stored as an array of 
> org.apache.falcon.entity.v0.cluster.Property, and cluster.getProperty("test") 
> will return either "value1" or "value2" but not both. If falcon does not 
> support multiple values for same property key,  parsing such an entity should 
> throw an error. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1343) validation of read/write endpoints is not reliable

2015-09-14 Thread Peeyush Bishnoi (JIRA)


[ 
https://issues.apache.org/jira/browse/FALCON-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744809#comment-14744809
 ] 

Peeyush Bishnoi commented on FALCON-1343:
-

[~bvellanki] I have added comment for a patch on review board. Please look.

> validation of read/write endpoints is not reliable
> --
>
> Key: FALCON-1343
> URL: https://issues.apache.org/jira/browse/FALCON-1343
> Project: Falcon
>  Issue Type: Sub-task
>  Components: general
>Reporter: Balu Vellanki
>Assignee: Balu Vellanki
> Attachments: FALCON-1343.patch
>
>
> A read/write endpoint is currently validated by creating a filesystem with 
> the endpoint url. 
> {code}
> HadoopClientFactory.get().createProxiedFileSystem(conf);
> {code}
> I confirmed with a HDFS team member that it is not sufficient validation.  
> Ideally check if the end user has access by doing atleast a list /tmp call 
> after creating proxiedFileSystem. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Discuss] : Should a non-superuser be allowed to update ACL of feed or process entity

2015-09-14 Thread Peeyush Bishnoi

Balu,
Thanks for initiating the discussion.
I am of the opinion here is that ACL of feed/process entity should work 
similarly to the UNIX-like system.
If user1 has not set the permission for group writable , then user2 should not 
be allowed to updateACL of feed or process entity. If user1 has set the 
permission for group writable purposefully, then user2 should alsoupdate as per 
the agreement between user1 and user2 (collaborative work) as they belong to 
same group.

Thanks,---Peeyush



 


 On Tuesday, 15 September 2015 10:23 AM, Sandeep Samudrala 
 wrote:
   

 I agree with above point to handle submission time. But again an entity can
be submitted and scheduled with different users, in which case the user
with which schedule is ran will be used. We might have to handle even
scheduling part. I think rather than handling ACL at various levels, the
whole ACL can be improved as part of FALCON-1367
.

On Tue, Sep 15, 2015 at 10:14 AM, pavan kumar Kolamuri <
pavan.kolam...@gmail.com> wrote:

> Even i agree that user2 shouldn't update/delete/suspend the entity, but we
> should be consistent across all API's for the same. As of now submit is
> allowed if user belongs to the same group of ACL owner group right ? Should
> we also change this behaviour to make sure only ACL owner should be allowed
> to submit ?
>
> On Tue, Sep 15, 2015 at 9:58 AM, Pallavi Rao 
> wrote:
>
> > Agree that "user2" shouldn't be allowed to just update the entity and
> > change the ownership. All the more reason to have a separate Auth API,
> > rather than embed the ACL in the entity itself. Such issues can be
> handled
> > in a much cleaner way.
> >
> > Regards,
> > Pallavi
> >
> > On Tue, Sep 15, 2015 at 3:12 AM, Balu Vellanki <
> bvella...@hortonworks.com>
> > wrote:
> >
> > > Hi Team,
> > >
> > > Today, Feed/Process entities have ACL with owner and group. Support for
> > > permissions is not implemented yet. Any user who is the owner OR who
> > > belongs to the group can update/delete/suspend the entity.
> > >
> > > If two users "user1" and "user2" belong to same group "users" and the
> > > falcon entity ACL is ,
> > > then user2 can update the falcon entity and claim ownership of this
> > entity.
> > > I believe that user2 should not be allowed to do so unless it is
> > > superuser.  Similar behavior is not allowed in HDFS.  Please comment if
> > you
> > > disagree.
> > >
> > > https://issues.apache.org/jira/browse/FALCON-1340
> > >
> > > Thanks
> > > Balu Velalnki
> > >
> >
> > --
> > _
> > The information contained in this communication is intended solely for
> the
> > use of the individual or entity to whom it is addressed and others
> > authorized to receive it. It may contain confidential or legally
> privileged
> > information. If you are not the intended recipient you are hereby
> notified
> > that any disclosure, copying, distribution or taking any action in
> reliance
> > on the contents of this information is strictly prohibited and may be
> > unlawful. If you have received this communication in error, please notify
> > us immediately by responding to this email and then delete it from your
> > system. The firm is neither liable for the proper and complete
> transmission
> > of the information contained in this communication nor for any delay in
> its
> > receipt.
> >
>
>
>
> --
> Regards
> Pavan Kumar Kolamuri
>

[jira] [Commented] (FALCON-1443) NPE in shutdown hook

2015-09-14 Thread sandeep samudrala (JIRA)


[ 
https://issues.apache.org/jira/browse/FALCON-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742989#comment-14742989
 ] 

sandeep samudrala commented on FALCON-1443:
---

Makes sense. +1. Looks good to me.

> NPE in shutdown hook
> 
>
> Key: FALCON-1443
> URL: https://issues.apache.org/jira/browse/FALCON-1443
> Project: Falcon
>  Issue Type: Bug
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
>Priority: Minor
> Fix For: trunk
>
> Attachments: FALCON-1443.patch
>
>
> When Falcon server fails to start because of invalid properties in 
> startup/runtime properties, the shutdown hook fails with a NPE:
> {code}
> 2015-09-11 10:29:35,702 INFO  - [main:] ~ Initializing 
> org.apache.falcon.util.StartupProperties properties with domain falcon 
> (ApplicationProperties:145)
> 2015-09-11 10:29:35,703 ERROR - [main:] ~ Key: prism.application.services 
> does not start with '*.' or 'falcon.' (ApplicationProperties:162)
> 2015-09-11 10:29:35,706 INFO  - [Thread-0:] ~ calling shutdown hook (Main:68)
> 2015-09-11 10:29:35,707 ERROR - [Thread-0:] ~ Server shutdown failed with  
> (Main:72)
> java.lang.NullPointerException
> at org.apache.falcon.Main$ShutDown.run(Main.java:69)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38294: FALCON-1434 Enhance schedule API to accept key-value properties



> On Sept. 11, 2015, 3:28 p.m., Balu Vellanki wrote:
> > common/src/main/java/org/apache/falcon/entity/EntityUtil.java, line 904
> > 
> >
> > Do you not allow ":" character in the value?  I see value in allowing 
> > ":" in the value for properties. Splitting kvPair into two Strings might be 
> > more helpful. 
> > 
> > String[] keyValue = kvPair.trim().split(":", 2);

Makes sense. Will allow : in values.


> On Sept. 11, 2015, 3:28 p.m., Balu Vellanki wrote:
> > common/src/main/java/org/apache/falcon/entity/EntityUtil.java, line 905
> > 
> >
> > This can allow the keyValue[0] to be empty. I think we should not have 
> > empty property keys.

Good catch. Added an additional check.


> On Sept. 11, 2015, 3:28 p.m., Balu Vellanki wrote:
> > common/src/test/java/org/apache/falcon/entity/EntityUtilTest.java, line 337
> > 
> >
> > Please add " :value1" to list of invalid props.

Done.


> On Sept. 11, 2015, 3:28 p.m., Balu Vellanki wrote:
> > client/src/main/java/org/apache/falcon/cli/FalconCLI.java, line 94
> > 
> >
> > Minor nit - Can we use "properties" instead?
> > 
> > Can we also add some tests to EntityManagerJerseyIT?

Changed to "properties".

Just enhanced the existing tests in FalconCLIIT and EntityManagerJerseyIT. 
Didn't add any new test as the properties aren't really used yet.


- Pallavi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38294/#review98613
---


On Sept. 14, 2015, 6:21 a.m., Pallavi Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38294/
> ---
> 
> (Updated Sept. 14, 2015, 6:21 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1434
> https://issues.apache.org/jira/browse/FALCON-1434
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> The schedule API will be enhanced to accept a key-value properties. This is a 
> foundation to enable users to specify the scheduler on which they want to 
> schedule the entity. This in turn enables migration to native scheduler from 
> Oozie.
> Example:
> bin/falcon entity -schedule -props falcon.scheduler=native -name 
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/falcon/cli/FalconCLI.java d4da302 
>   client/src/main/java/org/apache/falcon/client/AbstractFalconClient.java 
> 282b41b 
>   client/src/main/java/org/apache/falcon/client/FalconClient.java 44436d2 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 25d9008 
>   
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
>  ea86c2a 
>   common/src/test/java/org/apache/falcon/entity/EntityUtilTest.java cfdc84d 
>   
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java
>  5f79ca1 
>   
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java
>  f9405dc 
>   
> prism/src/main/java/org/apache/falcon/resource/proxy/SchedulableEntityManagerProxy.java
>  ceabb06 
>   unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java eb65cb3 
>   unit/src/test/java/org/apache/falcon/unit/FalconUnitTestBase.java 997b301 
>   unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java 498f50e 
>   
> webapp/src/main/java/org/apache/falcon/resource/SchedulableEntityManager.java 
> 1f8cc1b 
>   webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 0062070 
>   webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java 
> bcd3bd5 
>   webapp/src/test/java/org/apache/falcon/resource/TestContext.java 54671fb 
> 
> Diff: https://reviews.apache.org/r/38294/diff/
> 
> 
> Testing
> ---
> 
> UT added
> Manually tested to ensure CLI accepts properties and it is propagated.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>

[jira] [Created] (FALCON-1447) Integration Tests for native scheduler

Pallavi Rao created FALCON-1447:
---

 Summary: Integration Tests for native scheduler
 Key: FALCON-1447
 URL: https://issues.apache.org/jira/browse/FALCON-1447
 Project: Falcon
  Issue Type: Sub-task
Reporter: Pallavi Rao






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 35724: Base framework of the native scheduler

2015-09-14 Thread Srikanth Sundarrajan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35724/#review98194
---



common/src/main/java/org/apache/falcon/entity/EntityUtil.java (line 260)


Are we sure we want to calculate months this way ? There are other places 
in falcon this is done, but the usage mayn't have correctness implications, but 
being available in EntityUtil, might result in inadvertent-incorrect usage.



common/src/main/java/org/apache/falcon/entity/EntityUtil.java (line 634)


Is the staging dir owned by the entity submitting user ?



common/src/main/java/org/apache/falcon/entity/EntityUtil.java (line 641)


Second condition may suffice



common/src/main/java/org/apache/falcon/entity/EntityUtil.java (line 647)


Technically timestamp after md5 is a numeric value and is appended as is 
without any prefix padding, so it might not sort it correctly, but may not 
cause issues for a long time to come. Might be more readable as well to clearly 
compare on the FileStatus::getModificationTime() or comparing on the timestamp 
after numeric::parse



scheduler/pom.xml (line 57)


Why would we have oozie-adaptor dependency in Scheduler ?



scheduler/pom.xml (line 92)


Version should be setup in dependencyManagement in parent pom as a best 
practice



scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 
(line 38)


What is external ID ? JavaDoc would be useful. Do we also intend to keep a 
tracking url.



scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 
(line 39)


nominalTime is a Oozie hangover. Can we use instanceTime instead ?

Also is instanceTime/nominalTime mandatory for all scheduling types ? A 
Periodic ?



scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 
(line 104)


How do we handle re-runs in this world ? Will ExecutionInstance be 
different for different runs ?



scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 
(line 145)


EntityExecutor has these behaviors, why are they again in the Instance ? If 
Instance is modelled as a bean, then these behaviors can be avoided here.



scheduler/src/main/java/org/apache/falcon/execution/FalconExecutionService.java 
(line 46)


Should this be held in a ConcurrentMap<> instead of Map ?



scheduler/src/main/java/org/apache/falcon/execution/FalconExecutionService.java 
(line 53)


Why is StateStore::getEntities returning an iterable of EntityState ?



scheduler/src/main/java/org/apache/falcon/execution/NotificationHandler.java 
(line 33)


onEvent() or onNotification() might be more apt. Currently 
NotificationHandler seems to notify.



scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java
 (line 60)


Can we use DI here for the DAGEngine & ExecutionService ?

Also if we make ExecutionInstance a bean, these might be unnessacary and 
will be limited to EntityExecutor



scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java
 (line 109)


Cluster specific locations has to be considered



scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java
 (line 120)


DI ?



scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java
 (line 121)


Can't quite get where the feed EL's being evaluated and paths being 
resolved.



scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java
 (line 124)


Shouldn't be polling frequency be lot smaller than the feed frequency ?



scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java
 (line 126)


Not sure why data availability notification is being registered against 
ServicesRegistry.



scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java
 (line 127)

Re: Review Request 35724: Base framework of the native scheduler

2015-09-14 Thread Ajay Yadava

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35724/#review98751
---

common/src/main/java/org/apache/falcon/entity/EntityUtil.java (line 257)

This is incorrect, reason being that for certain frequencies like monthly
the time in millis between each run keeps changing. This will fail e.g. for
monthly feeds.

scheduler/pom.xml (line 92)

Any reasons to not use 2.2.1 which is the latest?

scheduler/pom.xml (line 103)

1.10.19 is the latest.

scheduler/pom.xml (line 109)

why not 2.8.2?

scheduler/src/main/java/org/apache/falcon/exception/DAGEngineException.java
(line 23)

Description seems to be incorrect.

scheduler/src/main/java/org/apache/falcon/notification/service/FalconNotificationService.java
(line 48)

Incorrect javadoc, it is actually for registering.

scheduler/src/main/java/org/apache/falcon/notification/service/event/DataEvent.java
(line 33)

numInstances and it's getter setters are not being used. Please remove it.
If you want for future then still delete it and file a JIRA.

scheduler/src/main/java/org/apache/falcon/notification/service/event/JobCompletedEvent.java
(line 28)

Remove stray todo statement from all classes.

scheduler/src/main/java/org/apache/falcon/notification/service/event/JobCompletedEvent.java
(line 37)

Constructor allows construction of incomplete object, which is not a good
practice.

scheduler/src/main/java/org/apache/falcon/notification/service/event/JobScheduledEvent.java
(line 30)

All constructors should form complete objects, i.e. all mandatory
parameters should be available after construction. Please make this change for
all the Event Classes

scheduler/src/main/java/org/apache/falcon/notification/service/request/DataNotificationRequest.java
(line 30)

Remove stray TODOs

scheduler/src/main/java/org/apache/falcon/notification/service/request/DataNotificationRequest.java
(line 37)

This field and it's getters and setters are unused. Please remove it.

scheduler/src/main/java/org/apache/falcon/notification/service/request/DataNotificationRequest.java
(line 43)

not used. Even if it is for future use cases then please remove it for now
and file a JIRA so that it doesn't get dropped.

scheduler/src/main/java/org/apache/falcon/notification/service/request/NotificationRequest.java
(line 38)

ORDER is not being set in any of the subclasses.

scheduler/src/main/java/org/apache/falcon/notification/service/request/TimeNotificationRequest.java
(line 31)

Builder pattern should not be used for supplying mandatory patterns as it
allows creation of incomplete objects. It should be used only for optional
parameters. Please change this from all NotificationRequest subclasses.

scheduler/src/main/java/org/apache/falcon/state/EntityState.java (line 71)

Incorrect message.

scheduler/src/main/java/org/apache/falcon/state/ID.java (line 33)

ID is overloaded for both entity key and instance key. Both should have
separate classes as the mandatory parameters for both of them are different.

scheduler/src/main/java/org/apache/falcon/state/ID.java (line 145)

toString seems to be used for InstanceKey this should be separated out.

scheduler/src/main/java/org/apache/falcon/state/ID.java (line 189)

This is incorrect. compareTo should be called only on objects of
same/comparable type and not on generic object. If you use the parameterized
version of " Comparator" this will give you compile time error.

- Ajay Yadava

On July 28, 2015, 11:07 a.m., Pallavi Rao wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35724/
> ---
>
> (Updated July 28, 2015, 11:07 a.m.)
>

[jira] [Commented] (FALCON-1213) Base framework of the native scheduler


[ 
https://issues.apache.org/jira/browse/FALCON-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744057#comment-14744057
 ] 

Ajay Yadava commented on FALCON-1213:
-

I have done a partial review, will finish the remaining review soon.

> Base framework of the native scheduler
> --
>
> Key: FALCON-1213
> URL: https://issues.apache.org/jira/browse/FALCON-1213
> Project: Falcon
>  Issue Type: Sub-task
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
> Attachments: ExecutionEngine.jpeg, FALCON-1213-v2.patch, 
> FALCON-1213.patch, NotificationServices.jpeg
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1401) MetadataMappingService fails to add an edge for a process instance


[ 
https://issues.apache.org/jira/browse/FALCON-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744090#comment-14744090
 ] 

Ajay Yadava commented on FALCON-1401:
-

Yes, it will be consistent but won't be backward compatible :) Also if you 
decide to go ahead with making it consistent then the value should still be 
"NONE" as that is more semantically correct :P 

It's a small change and I didn't mean to do nitpicking, I hope I have clarified 
the reasoning behind my suggestions.  Decide and proceed. :)

> MetadataMappingService fails to add an edge for a process instance
> --
>
> Key: FALCON-1401
> URL: https://issues.apache.org/jira/browse/FALCON-1401
> Project: Falcon
>  Issue Type: Bug
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
>Priority: Minor
> Attachments: FALCON-1401.patch
>
>
> If a process has no output feeds, the metadata mapping service fails to add 
> an edge for the process instances. It fails with NPE:
> {code}
> 2015-08-14 15:31:00,950 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:00,951 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,957 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:00,981 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,984 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,988 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,990 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,992 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,995 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,997 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,009 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:01,009 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,012 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:01,033 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,035 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,039 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,041 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,043 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,044 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,047 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,073 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:01,073 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,076 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
>

[jira] [Comment Edited] (FALCON-1401) MetadataMappingService fails to add an edge for a process instance


[ 
https://issues.apache.org/jira/browse/FALCON-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744090#comment-14744090
 ] 

Ajay Yadava edited comment on FALCON-1401 at 9/14/15 7:31 PM:
--

Yes, it will be consistent but won't be backward compatible :) Also if you 
decide to go ahead with making it consistent then the value should still be 
"NONE" as that is more semantically correct :P 

It's a small change and I didn't mean to do nitpicking, I hope I have clarified 
my suggestions.  Decide and proceed. :)


was (Author: ajayyadava):
Yes, it will be consistent but won't be backward compatible :) Also if you 
decide to go ahead with making it consistent then the value should still be 
"NONE" as that is more semantically correct :P 

It's a small change and I didn't mean to do nitpicking, I hope I have clarified 
the reasoning behind my suggestions.  Decide and proceed. :)

> MetadataMappingService fails to add an edge for a process instance
> --
>
> Key: FALCON-1401
> URL: https://issues.apache.org/jira/browse/FALCON-1401
> Project: Falcon
>  Issue Type: Bug
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
>Priority: Minor
> Attachments: FALCON-1401.patch
>
>
> If a process has no output feeds, the metadata mapping service fails to add 
> an edge for the process instances. It fails with NPE:
> {code}
> 2015-08-14 15:31:00,950 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:00,951 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,957 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:00,981 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,984 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,988 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,990 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,992 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,995 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,997 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,009 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:01,009 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,012 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:01,033 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,035 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,039 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,041 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,043 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,044 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,047 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,073 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z

[jira] [Commented] (FALCON-1443) NPE in shutdown hook


[ 
https://issues.apache.org/jira/browse/FALCON-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742990#comment-14742990
 ] 

Ajay Yadava commented on FALCON-1443:
-

Will commit shortly.

> NPE in shutdown hook
> 
>
> Key: FALCON-1443
> URL: https://issues.apache.org/jira/browse/FALCON-1443
> Project: Falcon
>  Issue Type: Bug
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
>Priority: Minor
> Fix For: trunk
>
> Attachments: FALCON-1443.patch
>
>
> When Falcon server fails to start because of invalid properties in 
> startup/runtime properties, the shutdown hook fails with a NPE:
> {code}
> 2015-09-11 10:29:35,702 INFO  - [main:] ~ Initializing 
> org.apache.falcon.util.StartupProperties properties with domain falcon 
> (ApplicationProperties:145)
> 2015-09-11 10:29:35,703 ERROR - [main:] ~ Key: prism.application.services 
> does not start with '*.' or 'falcon.' (ApplicationProperties:162)
> 2015-09-11 10:29:35,706 INFO  - [Thread-0:] ~ calling shutdown hook (Main:68)
> 2015-09-11 10:29:35,707 ERROR - [Thread-0:] ~ Server shutdown failed with  
> (Main:72)
> java.lang.NullPointerException
> at org.apache.falcon.Main$ShutDown.run(Main.java:69)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1434) Enhance schedule API to accept key-value properties


 [ 
https://issues.apache.org/jira/browse/FALCON-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pallavi Rao updated FALCON-1434:

Attachment: FALCON-1434-v2.patch

Review comments addressed.

> Enhance schedule API to accept key-value properties
> ---
>
> Key: FALCON-1434
> URL: https://issues.apache.org/jira/browse/FALCON-1434
> Project: Falcon
>  Issue Type: Sub-task
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
> Fix For: trunk
>
> Attachments: FALCON-1434-v2.patch, FALCON-1434.patch
>
>
> The schedule API will be enhanced to accept a key-value properties. This is a 
> foundation to enable users to specify the scheduler on which they want to 
> schedule the entity. This in turn enables migration to native scheduler from 
> Oozie.
> Example:
> bin/falcon entity -schedule -props falcon.scheduler=native -name 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1401) MetadataMappingService fails to add an edge for a process instance


[ 
https://issues.apache.org/jira/browse/FALCON-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743006#comment-14743006
 ] 

Pallavi Rao commented on FALCON-1401:
-

[~ajayyadava], I was looking at the patch that changed "NONE" to "IGNORE". The 
reason it is IGNORE (and not NONE) is that, there is an output feed, but, it 
might be writing to HCat and hence feed path/name must be ignored. In that 
light, I think both NONE and IGNORE are valid values that should be checked 
against. However, agree that FALCON-1091 should have changed the value to 
IGNORE conditionally (not always).

> MetadataMappingService fails to add an edge for a process instance
> --
>
> Key: FALCON-1401
> URL: https://issues.apache.org/jira/browse/FALCON-1401
> Project: Falcon
>  Issue Type: Bug
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
>Priority: Minor
> Attachments: FALCON-1401.patch
>
>
> If a process has no output feeds, the metadata mapping service fails to add 
> an edge for the process instances. It fails with NPE:
> {code}
> 2015-08-14 15:31:00,950 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:00,951 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,957 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:00,981 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,984 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,988 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,990 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,992 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:00,995 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:00,997 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,009 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:01,009 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,012 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:101)
> 2015-08-14 15:31:01,033 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process, type=PROCESS_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,035 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pig-sleep-process, type=PROCESS_ENTITY, v=v[16] 
> (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,039 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=local, type=CLUSTER_ENTITY (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,041 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=local, type=CLUSTER_ENTITY, v=v[4] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,043 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pallavi.rao, type=USER (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,044 INFO  - [ActiveMQ Session Task:] ~ Vertex exists? 
> name=pallavi.rao, type=USER, v=v[8] (InstanceRelationshipGraphBuilder:129)
> 2015-08-14 15:31:01,047 INFO  - [ActiveMQ Session Task:] ~ Computing feed 
> instance for : name= IGNORE path= IGNORE, in cluster: local 
> (InstanceRelationshipGraphBuilder:243)
> 2015-08-14 15:31:01,073 INFO  - [ActiveMQ Session Task:] ~ Adding process 
> instance: pig-sleep-process/2015-07-28T07:40Z 
> (InstanceRelationshipGraphBuilder:74)
> 2015-08-14 15:31:01,073 DEBUG - [ActiveMQ Session Task:] ~ Finding vertex 
> for: name=pig-sleep-process/2015-07-28T07:40Z, type=PROCESS_INSTANCE 
> (RelationshipGraphBuilder:87)
> 2015-08-14 15:31:01,076 DEBUG - [ActiveMQ Session Task:] ~ Creating a new 
> vertex for:

[jira] [Assigned] (FALCON-1446) Flaky TaskLogRetrieverYarnTest


 [ 
https://issues.apache.org/jira/browse/FALCON-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal reassigned FALCON-1446:
---

Assignee: Narayan Periwal

> Flaky TaskLogRetrieverYarnTest
> --
>
> Key: FALCON-1446
> URL: https://issues.apache.org/jira/browse/FALCON-1446
> Project: Falcon
>  Issue Type: Bug
>Reporter: Ajay Yadava
>Assignee: Narayan Periwal
>
> Currently the build fails arbitrarily. 
> An example failure is below.
>  java.lang.AssertionError: null: lists don't have the same size expected:<2> 
> but was:<9>
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.failNotEquals(Assert.java:489)
> at org.testng.Assert.assertEquals(Assert.java:118)
> at org.testng.Assert.assertEquals(Assert.java:365)
> at org.testng.Assert.assertEquals(Assert.java:532)
> at org.testng.Assert.assertEquals(Assert.java:509)
> at 
> org.apache.falcon.oozie.logging.TaskLogRetrieverYarnTest.testSuccess(TaskLogRetrieverYarnTest.java:90)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38294: FALCON-1434 Enhance schedule API to accept key-value properties


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38294/
---

(Updated Sept. 14, 2015, 6:21 a.m.)


Review request for Falcon.


Changes
---

Addressed review comments


Bugs: FALCON-1434
https://issues.apache.org/jira/browse/FALCON-1434


Repository: falcon-git


Description
---

The schedule API will be enhanced to accept a key-value properties. This is a 
foundation to enable users to specify the scheduler on which they want to 
schedule the entity. This in turn enables migration to native scheduler from 
Oozie.
Example:
bin/falcon entity -schedule -props falcon.scheduler=native -name 


Diffs (updated)
-

  client/src/main/java/org/apache/falcon/cli/FalconCLI.java d4da302 
  client/src/main/java/org/apache/falcon/client/AbstractFalconClient.java 
282b41b 
  client/src/main/java/org/apache/falcon/client/FalconClient.java 44436d2 
  common/src/main/java/org/apache/falcon/entity/EntityUtil.java 25d9008 
  
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
 ea86c2a 
  common/src/test/java/org/apache/falcon/entity/EntityUtilTest.java cfdc84d 
  
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 
5f79ca1 
  
prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java
 f9405dc 
  
prism/src/main/java/org/apache/falcon/resource/proxy/SchedulableEntityManagerProxy.java
 ceabb06 
  unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java eb65cb3 
  unit/src/test/java/org/apache/falcon/unit/FalconUnitTestBase.java 997b301 
  unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java 498f50e 
  webapp/src/main/java/org/apache/falcon/resource/SchedulableEntityManager.java 
1f8cc1b 
  webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 0062070 
  webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java 
bcd3bd5 
  webapp/src/test/java/org/apache/falcon/resource/TestContext.java 54671fb 

Diff: https://reviews.apache.org/r/38294/diff/


Testing
---

UT added
Manually tested to ensure CLI accepts properties and it is propagated.


Thanks,

Pallavi Rao

[jira] [Created] (FALCON-1446) Flaky TaskLogRetrieverYarnTest

Ajay Yadava created FALCON-1446:
---

 Summary: Flaky TaskLogRetrieverYarnTest
 Key: FALCON-1446
 URL: https://issues.apache.org/jira/browse/FALCON-1446
 Project: Falcon
  Issue Type: Bug
Reporter: Ajay Yadava


Currently the build fails arbitrarily. 

An example failure is below.
 java.lang.AssertionError: null: lists don't have the same size expected:<2> 
but was:<9>
at org.testng.Assert.fail(Assert.java:89)
at org.testng.Assert.failNotEquals(Assert.java:489)
at org.testng.Assert.assertEquals(Assert.java:118)
at org.testng.Assert.assertEquals(Assert.java:365)
at org.testng.Assert.assertEquals(Assert.java:532)
at org.testng.Assert.assertEquals(Assert.java:509)
at 
org.apache.falcon.oozie.logging.TaskLogRetrieverYarnTest.testSuccess(TaskLogRetrieverYarnTest.java:90)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1446) Flaky TaskLogRetrieverYarnTest


 [ 
https://issues.apache.org/jira/browse/FALCON-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal updated FALCON-1446:

Attachment: FALCON-1446-v0.patch

> Flaky TaskLogRetrieverYarnTest
> --
>
> Key: FALCON-1446
> URL: https://issues.apache.org/jira/browse/FALCON-1446
> Project: Falcon
>  Issue Type: Bug
>Reporter: Ajay Yadava
>Assignee: Narayan Periwal
> Attachments: FALCON-1446-v0.patch
>
>
> Currently the build fails arbitrarily. 
> An example failure is below.
>  java.lang.AssertionError: null: lists don't have the same size expected:<2> 
> but was:<9>
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.failNotEquals(Assert.java:489)
> at org.testng.Assert.assertEquals(Assert.java:118)
> at org.testng.Assert.assertEquals(Assert.java:365)
> at org.testng.Assert.assertEquals(Assert.java:532)
> at org.testng.Assert.assertEquals(Assert.java:509)
> at 
> org.apache.falcon.oozie.logging.TaskLogRetrieverYarnTest.testSuccess(TaskLogRetrieverYarnTest.java:90)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-1233) Migration from oozie coord and native scheduler


 [ 
https://issues.apache.org/jira/browse/FALCON-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pallavi Rao updated FALCON-1233:

Attachment: FALCON-1233.patch

The patch needs FALCON-1213, FALCON-1434. Cannot upload to Review Board until 
the required patches are committed.

> Migration from oozie coord and native scheduler
> ---
>
> Key: FALCON-1233
> URL: https://issues.apache.org/jira/browse/FALCON-1233
> Project: Falcon
>  Issue Type: Sub-task
>Reporter: Pallavi Rao
>Assignee: Pallavi Rao
> Attachments: FALCON-1233.patch, Migrating to Native Scheduler - 
> Approach v2.pdf, Migrating to Native Scheduler - Approach.pdf
>
>
> The migration should be as seamless as possible and the user should be able 
> to migrate to the native scheduler in a phased manner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-503) The expected result of examples/data/generate.sh <>?


[ 
https://issues.apache.org/jira/browse/FALCON-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743165#comment-14743165
 ] 

Narayan Periwal commented on FALCON-503:


The examples/data/generate.sh script is used to generate the input data for the 
falcon processes(that are there inside examples/entity/filesystem/) to consume.

The script will generate the /data directory in the hdfs which will contain the 
input data, as depicted below:-

[oozie@/opt/gridops/falcon$]hadoop fs -ls /data/in/2013/11/15/00
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

15/09/14 14:11:49 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
Found 60 items
drwxr-xr-x   - oozie hdfs  0 2015-09-02 18:12 /data/in/2013/11/15/00/00
drwxr-xr-x   - oozie hdfs  0 2015-09-02 18:12 /data/in/2013/11/15/00/01
drwxr-xr-x   - oozie hdfs  0 2015-09-02 18:12 /data/in/2013/11/15/00/02
drwxr-xr-x   - oozie hdfs  0 2015-09-02 18:12 /data/in/2013/11/15/00/03
drwxr-xr-x   - oozie hdfs  0 2015-09-02 18:12 /data/in/2013/11/15/00/04
drwxr-xr-x   - oozie hdfs  0 2015-09-02 18:12 /data/in/2013/11/15/00/05
drwxr-xr-x   - oozie hdfs  0 2015-09-02 18:12 /data/in/2013/11/15/00/06
drwxr-xr-x   - oozie hdfs  0 2015-09-02 18:12 /data/in/2013/11/15/00/07
 ...

> The expected result of examples/data/generate.sh <>?
> ---
>
> Key: FALCON-503
> URL: https://issues.apache.org/jira/browse/FALCON-503
> Project: Falcon
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 0.6
>Reporter: Peng
>
> In the current version setup, there is a command examples/data/generate.sh 
> <>. What is the expected result of this command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FALCON-1446) Flaky TaskLogRetrieverYarnTest


[ 
https://issues.apache.org/jira/browse/FALCON-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743189#comment-14743189
 ] 

Pallavi Rao commented on FALCON-1446:
-

+1 for the patch. Will commit unless anyone has any comments.

> Flaky TaskLogRetrieverYarnTest
> --
>
> Key: FALCON-1446
> URL: https://issues.apache.org/jira/browse/FALCON-1446
> Project: Falcon
>  Issue Type: Bug
>Reporter: Ajay Yadava
>Assignee: Narayan Periwal
> Attachments: FALCON-1446-v0.patch
>
>
> Currently the build fails arbitrarily. 
> An example failure is below.
>  java.lang.AssertionError: null: lists don't have the same size expected:<2> 
> but was:<9>
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.failNotEquals(Assert.java:489)
> at org.testng.Assert.assertEquals(Assert.java:118)
> at org.testng.Assert.assertEquals(Assert.java:365)
> at org.testng.Assert.assertEquals(Assert.java:532)
> at org.testng.Assert.assertEquals(Assert.java:509)
> at 
> org.apache.falcon.oozie.logging.TaskLogRetrieverYarnTest.testSuccess(TaskLogRetrieverYarnTest.java:90)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FALCON-503) The expected result of examples/data/generate.sh <>?


 [ 
https://issues.apache.org/jira/browse/FALCON-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal updated FALCON-503:
---
Assignee: (was: Narayan Periwal)

> The expected result of examples/data/generate.sh <>?
> ---
>
> Key: FALCON-503
> URL: https://issues.apache.org/jira/browse/FALCON-503
> Project: Falcon
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 0.6
>Reporter: Peng
>
> In the current version setup, there is a command examples/data/generate.sh 
> <>. What is the expected result of this command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FALCON-503) The expected result of examples/data/generate.sh <>?


 [ 
https://issues.apache.org/jira/browse/FALCON-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal reassigned FALCON-503:
--

Assignee: Narayan Periwal

> The expected result of examples/data/generate.sh <>?
> ---
>
> Key: FALCON-503
> URL: https://issues.apache.org/jira/browse/FALCON-503
> Project: Falcon
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 0.6
>Reporter: Peng
>Assignee: Narayan Periwal
>
> In the current version setup, there is a command examples/data/generate.sh 
> <>. What is the expected result of this command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38346: FALCON-1446: Flaky TaskLogRetrieverYarnTest


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38346/#review98826
---

Ship it!


Ship It!

- Pallavi Rao


On Sept. 14, 2015, 8:08 a.m., Narayan Periwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38346/
> ---
> 
> (Updated Sept. 14, 2015, 8:08 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Repository: falcon-git
> 
> 
> Description
> ---
> 
> Currently the build fails arbitrarily. 
> 
> An example failure is below.
>  java.lang.AssertionError: null: lists don't have the same size expected:<2> 
> but was:<9>
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.failNotEquals(Assert.java:489)
> at org.testng.Assert.assertEquals(Assert.java:118)
> at org.testng.Assert.assertEquals(Assert.java:365)
> at org.testng.Assert.assertEquals(Assert.java:532)
> at org.testng.Assert.assertEquals(Assert.java:509)
> at 
> org.apache.falcon.oozie.logging.TaskLogRetrieverYarnTest.testSuccess(TaskLogRetrieverYarnTest.java:90)
> 
> 
> Diffs
> -
> 
>   
> oozie/src/test/java/org/apache/falcon/oozie/logging/TaskLogRetrieverYarnTest.java
>  0d9cf12 
> 
> Diff: https://reviews.apache.org/r/38346/diff/
> 
> 
> Testing
> ---
> 
> Done.
> 
> 
> Thanks,
> 
> Narayan Periwal
> 
>

[jira] [Commented] (FALCON-1401) MetadataMappingService fails to add an edge for a process instance