[
https://issues.apache.org/jira/browse/FALCON-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797715#comment-13797715
]
Srikanth Sundarrajan edited comment on FALCON-129 at 10/17/13 2:44 PM:
-----------------------------------------------------------------------
-Patch doesn't apply. Can you please fix this?-
{code}
sriksun:falcon-for-commit sriksun$ curl
"https://issues.apache.org/jira/secure/attachment/12608821/FALCON-129-r2-cumulative.patch";
| git apply --check -v
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 65944 100 65944 0 0 24206 0 0:00:02 0:00:02 --:--:-- 27170
Checking patch
client/src/main/java/org/apache/falcon/client/FalconClient.java...
Checking patch
common/src/main/java/org/apache/falcon/catalog/CatalogPartition.java...
error: common/src/main/java/org/apache/falcon/catalog/CatalogPartition.java: No
such file or directory
Checking patch
common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java...
error: while searching for:
protected void delete(Cluster cluster, Entity entity, long retention,
FileStatus[] logs)
throws FalconException {
long now = System.currentTimeMillis();
error: patch failed:
common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java:106
error:
common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java:
patch does not apply
Checking patch common/src/main/java/org/apache/falcon/entity/EntityUtil.java...
Checking patch common/src/main/java/org/apache/falcon/entity/FeedHelper.java...
Hunk #1 succeeded at 145 (offset -10 lines).
Checking patch
common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java...
Hunk #1 succeeded at 74 (offset 2 lines).
Checking patch
common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java...
Checking patch
common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java...
Hunk #1 succeeded at 172 (offset -3 lines).
error: while searching for:
}
}
if (process.getLateProcess() != null) {
for (LateInput lp : process.getLateProcess().getLateInputs()) {
if (!feeds.keySet().contains(lp.getInput())) {
throw new ValidationException("Late Input: " +
lp.getInput() + " is not specified in the inputs");
}
try {
Feed feed = ConfigurationStore.get().get(EntityType.FEED,
feeds.get(lp.getInput()));
if (feed.getLateArrival() == null) {
throw new ValidationException(
"Late Input feed: " + lp.getInput() + " is not
configured with late arrival cut-off");
}
} catch (FalconException e) {
throw new ValidationException(e);
}
}
}
}
private void validateOptionalInputsForTableStorage(Feed feed, Input input)
throws FalconException {
if (input.isOptional() && FeedHelper.createStorage(feed).getType() ==
Storage.TYPE.TABLE) {
throw new ValidationException("Optional Input is not supported for
feeds with table storage! "
+ input.getName());
}
error: patch failed:
common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java:182
error:
common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java:
patch does not apply
Checking patch
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java...
Checking patch common/src/test/resources/config/feed/hive-table-feed.xml...
Checking patch
feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java...
error: while searching for:
} else if (sourceStorage.getType() == Storage.TYPE.TABLE) {
instancePaths = "${coord:dataIn('input')}";
props.put("shouldRecord", "false"); // todo - override
until late data is handled
final CatalogStorage sourceTableStorage = (CatalogStorage)
sourceStorage;
propagateTableStorageProperties(srcCluster,
sourceTableStorage, props, "falconSource");
final CatalogStorage targetTableStorage = (CatalogStorage)
targetStorage;
error: patch failed:
feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java:416
error: feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java:
patch does not apply
Checking patch
feed/src/main/resources/config/workflow/replication-workflow.xml...
error: while searching for:
<arg>${falconInPaths}</arg>
<arg>-falconInputFeeds</arg>
<arg>${falconInputFeeds}</arg>
<capture-output/>
</java>
<ok to="replication-decision"/>
error: patch failed:
feed/src/main/resources/config/workflow/replication-workflow.xml:46
error: feed/src/main/resources/config/workflow/replication-workflow.xml: patch
does not apply
Checking patch feed/src/main/resources/config/workflow/retention-workflow.xml...
Hunk #1 succeeded at 96 (offset -11 lines).
Checking patch
messaging/src/main/java/org/apache/falcon/messaging/EntityInstanceMessage.java...
Checking patch
messaging/src/main/java/org/apache/falcon/messaging/MessageProducer.java...
Checking patch
messaging/src/test/java/org/apache/falcon/messaging/FalconTopicProducerTest.java...
Checking patch
messaging/src/test/java/org/apache/falcon/messaging/FeedProducerTest.java...
Checking patch
messaging/src/test/java/org/apache/falcon/messaging/ProcessProducerTest.java...
Checking patch
oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java...
Checking patch
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java...
Checking patch
oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java...
Checking patch
prism/src/main/java/org/apache/falcon/service/FalconTopicSubscriber.java...
Checking patch
process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java...
error: while searching for:
import org.apache.falcon.entity.FeedHelper;
import org.apache.falcon.entity.ProcessHelper;
import org.apache.falcon.entity.Storage;
import org.apache.falcon.entity.v0.EntityType;
import org.apache.falcon.entity.v0.Frequency;
import org.apache.falcon.entity.v0.SchemaHelper;
error: patch failed:
process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java:27
error:
process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java:
patch does not apply
Checking patch
process/src/main/resources/config/workflow/process-parent-workflow.xml...
Hunk #2 succeeded at 140 (offset -9 lines).
Hunk #3 succeeded at 209 (offset -9 lines).
Checking patch
process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java...
Hunk #1 succeeded at 40 (offset -5 lines).
Hunk #2 succeeded at 155 (offset -9 lines).
error: while searching for:
props.put(prefix + "_partition_filter_java",
"${coord:dataInPartitionFilter('input', 'java')}");
} else if (prefix.equals("output")) {
props.put(prefix + "_dataout_partitions",
"${coord:dataOutPartitions('output')}");
props.put("shouldRecord", "false"); // todo - override until late
data is handled
}
}
error: patch failed:
process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java:285
error:
process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java:
patch does not apply
Checking patch process/src/test/resources/config/process/process-0.1.xml...
Checking patch
rerun/src/main/java/org/apache/falcon/latedata/LateDataHandler.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/event/LaterunEvent.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/event/RerunEvent.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/event/RerunEventFactory.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/event/RetryEvent.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunConsumer.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java...
Checking patch
rerun/src/test/java/org/apache/falcon/rerun/queue/ActiveMQTest.java...
Checking patch
webapp/src/test/java/org/apache/falcon/resource/FeedEntityValidationIT.java...
Checking patch webapp/src/test/resources/hive-table-feed.xml...
{code}
was (Author: sriksun):
Patch doesn't apply. Can you please fix this?
{code}
sriksun:falcon-for-commit sriksun$ curl
"https://issues.apache.org/jira/secure/attachment/12608821/FALCON-129-r2-cumulative.patch";
| git apply --check -v
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 65944 100 65944 0 0 24206 0 0:00:02 0:00:02 --:--:-- 27170
Checking patch
client/src/main/java/org/apache/falcon/client/FalconClient.java...
Checking patch
common/src/main/java/org/apache/falcon/catalog/CatalogPartition.java...
error: common/src/main/java/org/apache/falcon/catalog/CatalogPartition.java: No
such file or directory
Checking patch
common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java...
error: while searching for:
protected void delete(Cluster cluster, Entity entity, long retention,
FileStatus[] logs)
throws FalconException {
long now = System.currentTimeMillis();
error: patch failed:
common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java:106
error:
common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java:
patch does not apply
Checking patch common/src/main/java/org/apache/falcon/entity/EntityUtil.java...
Checking patch common/src/main/java/org/apache/falcon/entity/FeedHelper.java...
Hunk #1 succeeded at 145 (offset -10 lines).
Checking patch
common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java...
Hunk #1 succeeded at 74 (offset 2 lines).
Checking patch
common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java...
Checking patch
common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java...
Hunk #1 succeeded at 172 (offset -3 lines).
error: while searching for:
}
}
if (process.getLateProcess() != null) {
for (LateInput lp : process.getLateProcess().getLateInputs()) {
if (!feeds.keySet().contains(lp.getInput())) {
throw new ValidationException("Late Input: " +
lp.getInput() + " is not specified in the inputs");
}
try {
Feed feed = ConfigurationStore.get().get(EntityType.FEED,
feeds.get(lp.getInput()));
if (feed.getLateArrival() == null) {
throw new ValidationException(
"Late Input feed: " + lp.getInput() + " is not
configured with late arrival cut-off");
}
} catch (FalconException e) {
throw new ValidationException(e);
}
}
}
}
private void validateOptionalInputsForTableStorage(Feed feed, Input input)
throws FalconException {
if (input.isOptional() && FeedHelper.createStorage(feed).getType() ==
Storage.TYPE.TABLE) {
throw new ValidationException("Optional Input is not supported for
feeds with table storage! "
+ input.getName());
}
error: patch failed:
common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java:182
error:
common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java:
patch does not apply
Checking patch
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java...
Checking patch common/src/test/resources/config/feed/hive-table-feed.xml...
Checking patch
feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java...
error: while searching for:
} else if (sourceStorage.getType() == Storage.TYPE.TABLE) {
instancePaths = "${coord:dataIn('input')}";
props.put("shouldRecord", "false"); // todo - override
until late data is handled
final CatalogStorage sourceTableStorage = (CatalogStorage)
sourceStorage;
propagateTableStorageProperties(srcCluster,
sourceTableStorage, props, "falconSource");
final CatalogStorage targetTableStorage = (CatalogStorage)
targetStorage;
error: patch failed:
feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java:416
error: feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java:
patch does not apply
Checking patch
feed/src/main/resources/config/workflow/replication-workflow.xml...
error: while searching for:
<arg>${falconInPaths}</arg>
<arg>-falconInputFeeds</arg>
<arg>${falconInputFeeds}</arg>
<capture-output/>
</java>
<ok to="replication-decision"/>
error: patch failed:
feed/src/main/resources/config/workflow/replication-workflow.xml:46
error: feed/src/main/resources/config/workflow/replication-workflow.xml: patch
does not apply
Checking patch feed/src/main/resources/config/workflow/retention-workflow.xml...
Hunk #1 succeeded at 96 (offset -11 lines).
Checking patch
messaging/src/main/java/org/apache/falcon/messaging/EntityInstanceMessage.java...
Checking patch
messaging/src/main/java/org/apache/falcon/messaging/MessageProducer.java...
Checking patch
messaging/src/test/java/org/apache/falcon/messaging/FalconTopicProducerTest.java...
Checking patch
messaging/src/test/java/org/apache/falcon/messaging/FeedProducerTest.java...
Checking patch
messaging/src/test/java/org/apache/falcon/messaging/ProcessProducerTest.java...
Checking patch
oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java...
Checking patch
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java...
Checking patch
oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java...
Checking patch
prism/src/main/java/org/apache/falcon/service/FalconTopicSubscriber.java...
Checking patch
process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java...
error: while searching for:
import org.apache.falcon.entity.FeedHelper;
import org.apache.falcon.entity.ProcessHelper;
import org.apache.falcon.entity.Storage;
import org.apache.falcon.entity.v0.EntityType;
import org.apache.falcon.entity.v0.Frequency;
import org.apache.falcon.entity.v0.SchemaHelper;
error: patch failed:
process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java:27
error:
process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java:
patch does not apply
Checking patch
process/src/main/resources/config/workflow/process-parent-workflow.xml...
Hunk #2 succeeded at 140 (offset -9 lines).
Hunk #3 succeeded at 209 (offset -9 lines).
Checking patch
process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java...
Hunk #1 succeeded at 40 (offset -5 lines).
Hunk #2 succeeded at 155 (offset -9 lines).
error: while searching for:
props.put(prefix + "_partition_filter_java",
"${coord:dataInPartitionFilter('input', 'java')}");
} else if (prefix.equals("output")) {
props.put(prefix + "_dataout_partitions",
"${coord:dataOutPartitions('output')}");
props.put("shouldRecord", "false"); // todo - override until late
data is handled
}
}
error: patch failed:
process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java:285
error:
process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java:
patch does not apply
Checking patch process/src/test/resources/config/process/process-0.1.xml...
Checking patch
rerun/src/main/java/org/apache/falcon/latedata/LateDataHandler.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/event/LaterunEvent.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/event/RerunEvent.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/event/RerunEventFactory.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/event/RetryEvent.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunConsumer.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java...
Checking patch
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java...
Checking patch
rerun/src/test/java/org/apache/falcon/rerun/queue/ActiveMQTest.java...
Checking patch
webapp/src/test/java/org/apache/falcon/resource/FeedEntityValidationIT.java...
Checking patch webapp/src/test/resources/hive-table-feed.xml...
{code}
> Disable Late data handling for hive tables
> ------------------------------------------
>
> Key: FALCON-129
> URL: https://issues.apache.org/jira/browse/FALCON-129
> Project: Falcon
> Issue Type: Sub-task
> Affects Versions: 0.3
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Attachments: FALCON-129.patch, FALCON-129-r1.patch,
> FALCON-129-r2-cumulative.patch
>
>
> HCat nor Hive APIs expose internal stats about a given partition. The only
> way to get the partition size is to get the location of the partition on HDFS
> and then use globStatus and contentSummary APIs.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
