Hi Team,
We were trying to run an Import job with Apache Falcon. The document says
that
- [incremental, append] - data is extracted incrementally using the key
specified in the deltacolumn is supported
<import>
<source name="mysql-db" tableName="simple">
<extract type="incremental">
<deltacolumn>modified_time</deltacolumn>
<mergepolicy>append</mergepolicy>
</extract>
<fields>
<includes>
<field>id</field>
<field>name</field>
</includes>
</fields>
</source>
<arguments>
<argument name="--split-by" value="id"/>
<argument name="--num-mappers" value="2"/>
</arguments>
</import>
Whille [incremental, snapshot] - this is not supported.
We were trying to work with the former one and suprisingly when we submit
the feed we get an Error
ERROR: Bad Request;default/Feed feed-employee2 is using unsupported
extraction mechanism incremental
Log:
2016-06-01 08:04:24,761 INFO - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
All locks released on feed-employee2 (AbstractEntityManager:371)
2016-06-01 08:04:24,762 ERROR - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
Unable to persist entity object (AbstractEntityManager:194)
org.apache.falcon.entity.parser.ValidationException: Feed feed-employee2 is
using unsupported extraction mechanism incremental
at
org.apache.falcon.entity.parser.FeedEntityParser.validateFeedExtractionType(FeedEntityParser.java:595)
at
org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:105)
at
org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:71)
at
org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:519)
at
org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:443)
at
org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:191)
at
org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$2.doExecute(SchedulableEntityManagerProxy.java:177)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:773)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody2(SchedulableEntityManagerProxy.java:179)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure3.run(SchedulableEntityManagerProxy.java:1)
at
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at
org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:157)
at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)
at
org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2016-06-01 08:04:24,763 INFO - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
{Action:submit, Dimensions:{entityType=feed, colo=NULL}, Status: FAILED,
Time-taken:11824449 ns} (METRIC:38)
2016-06-01 08:04:24,763 ERROR - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
Exception in sending Notification from
EmailNotificationPlugin:java.lang.NullPointerException
(EmailNotificationPlugin:71)
2016-06-01 08:04:24,764 DEBUG - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop/127.0.0.1 performed
request https://localhost:15443/api/entities/submit/feed (127.0.0.1) at
time 2016-06-01T08:04Z (FalconAuditFilter:86)
2016-06-01 08:04:24,764 INFO - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: [email protected] performed
https://localhost:15443/api/entities/submit/feed (127.0.0.1) at
2016-06-01T08:04Z (AUDIT:48
Should this error come up.
PLease have a look at the submitted feed too.
<?xml version="1.0" encoding="UTF-8"?>
<!--
A feed representing Hourly customer email data retained for 90 days
-->
<feed description="Raw customer email feed" name="feed1"
xmlns="uri:falcon:feed:0.1">
<tags>externalSystem=USWestEmailServers,classification=secure</tags>
<groups>DataImportPipeline</groups>
<frequency>minutes(6)</frequency>
<late-arrival cut-off="hours(4)"/>
<clusters>
<cluster name="primaryCluster" type="source">
<validity start="2016-05-23T00:00Z"
end="2017-05-31T00:00Z"/>
<retention limit="days(90)" action="delete"/>
<import>
<source name="mysql-db" tableName="employee">
<extract type="incremental">
<deltacolumn>modified_item</deltacolumn>
<mergepolicy>append</mergepolicy>
</extract>
</source>
<arguments>
<argument name="--split-by" value="id"/>
<argument name="--num-mappers" value="2"/>
</arguments>
</import>
</cluster>
</clusters>
<locations>
<location type="data"
path="/falcon/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}"/>
<location type="stats" path="/none"/>
<location type="meta" path="/none"/>
</locations>
<notification type="email" to="xyz.com"/>
<ACL owner="hadoop" group="hadoop" permission="0755"/>
<schema location="/none" provider="none"/>
</feed>
