[jira] [Commented] (FALCON-95) Enable embedding hive scripts directly in a process

[Venkatesh Seetharam (JIRA)]

    [ 
https://issues.apache.org/jira/browse/FALCON-95?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809302#comment-13809302
 ] 

Venkatesh Seetharam commented on FALCON-95:
-------------------------------------------

Thanks [~samarthg] for running the tests. I wonder how since the code takes 
care of these:

Location is fetched from Cluster if any else then from Feed.
{code}
    public static Storage createStorage(Cluster cluster, Feed feed,
                                        
org.apache.falcon.entity.v0.cluster.Cluster clusterEntity)
        throws FalconException {

        final List<Location> locations = getLocations(cluster, feed);
        if (locations != null) {
            return new FileSystemStorage(clusterEntity, feed);
        }
....
    private static List<Location> getLocations(Cluster cluster, Feed feed) {
        // check if locations are overridden in cluster
        final Locations clusterLocations = cluster.getLocations();
        if (clusterLocations != null
                && clusterLocations.getLocations().size() != 0) {
            return clusterLocations.getLocations();
        }

        final Locations feedLocations = feed.getLocations();
        return feedLocations == null ? null : feedLocations.getLocations();
    }
{code}

The source and target paths are defined here.
{code}
        private void propagateFileSystemCopyProperties(String 
pathsWithPartitions,
                                                       Map<String, String> 
props) throws FalconException {
            String parts = pathsWithPartitions.replaceAll("//+", "/");
            parts = StringUtils.stripEnd(parts, "/");
            props.put("sourceRelativePaths", parts);

            props.put("distcpSourcePaths", "${coord:dataIn('input')}");
            props.put("distcpTargetPaths", "${coord:dataOut('output')}");
        }
{code}

Let me write more unit tests to see if I can find the bug. Thanks again for 
your time.

> Enable embedding hive scripts directly in a process
> ---------------------------------------------------
>
>                 Key: FALCON-95
>                 URL: https://issues.apache.org/jira/browse/FALCON-95
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.3
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>         Attachments: FALCON-95-doc-update.patch, FALCON-95.patch, 
> FALCON-95-r0.patch, FALCON-95-r1.patch, feed.xml, logs.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)