Hi Team, It seems the code could have some errors..
This is the method that throws the error.. The method says that incremental and Mergetype append is supported but then according to this Incremental extract will always throw the exception. Am I looking into it correct? Please advise [cid:[email protected]] Kind Regards Jainendra [unnamed] Go Green From: Jainendra Rai [mailto:[email protected]] Sent: Wednesday, June 01, 2016 6:56 PM To: Rai, Jainendra Subject: Fwd: Falcon Incremental Import support not available Best Regards, +91-9741823970 [email protected]<mailto:[email protected]> ---------- Forwarded message ---------- From: Jainendra Rai <[email protected]<mailto:[email protected]>> Date: Wed, Jun 1, 2016 at 1:41 PM Subject: Falcon Incremental Import support not available To: [email protected]<mailto:[email protected]> Hi Team, We were trying to run an Import job with Apache Falcon. The document says that ยท [incremental, append] - data is extracted incrementally using the key specified in the deltacolumn is supported <import> <source name="mysql-db" tableName="simple"> <extract type="incremental"> <deltacolumn>modified_time</deltacolumn> <mergepolicy>append</mergepolicy> </extract> <fields> <includes> <field>id</field> <field>name</field> </includes> </fields> </source> <arguments> <argument name="--split-by" value="id"/> <argument name="--num-mappers" value="2"/> </arguments> </import> Whille [incremental, snapshot] - this is not supported. We were trying to work with the former one and suprisingly when we submit the feed we get an Error ERROR: Bad Request;default/Feed feed-employee2 is using unsupported extraction mechanism incremental Log: 2016-06-01 08:04:24,761 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ All locks released on feed-employee2 (AbstractEntityManager:371) 2016-06-01 08:04:24,762 ERROR - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ Unable to persist entity object (AbstractEntityManager:194) org.apache.falcon.entity.parser.ValidationException: Feed feed-employee2 is using unsupported extraction mechanism incremental at org.apache.falcon.entity.parser.FeedEntityParser.validateFeedExtractionType(FeedEntityParser.java:595) at org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:105) at org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:71) at org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:519) at org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:443) at org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:191) at org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49) at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$2.doExecute(SchedulableEntityManagerProxy.java:177) at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:773) at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody2(SchedulableEntityManagerProxy.java:179) at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure3.run(SchedulableEntityManagerProxy.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51) at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:157) at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554) at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2016-06-01 08:04:24,763 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ {Action:submit, Dimensions:{entityType=feed, colo=NULL}, Status: FAILED, Time-taken:11824449 ns} (METRIC:38) 2016-06-01 08:04:24,763 ERROR - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ Exception in sending Notification from EmailNotificationPlugin:java.lang.NullPointerException (EmailNotificationPlugin:71) 2016-06-01 08:04:24,764 DEBUG - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop/127.0.0.1<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=AuZ2Uqn6QXWYp_ESwW_1AMsG9fhQCT1olHCBR6HPoRg&e=> performed request https://localhost:15443/api/entities/submit/feed<https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A15443_api_entities_submit_feed&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=jKPtKEZZX76LCp06hGh_Vlv4AbotHIotULACNZmpbUE&e=> (127.0.0.1) at time 2016-06-01T08:04Z (FalconAuditFilter:86) 2016-06-01 08:04:24,764 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: [email protected]<mailto:[email protected]> performed https://localhost:15443/api/entities/submit/feed<https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A15443_api_entities_submit_feed&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=jKPtKEZZX76LCp06hGh_Vlv4AbotHIotULACNZmpbUE&e=> (127.0.0.1) at 2016-06-01T08:04Z (AUDIT:48 Should this error come up. PLease have a look at the submitted feed too. <?xml version="1.0" encoding="UTF-8"?> <!-- A feed representing Hourly customer email data retained for 90 days --> <feed description="Raw customer email feed" name="feed1" xmlns="uri:falcon:feed:0.1"> <tags>externalSystem=USWestEmailServers,classification=secure</tags> <groups>DataImportPipeline</groups> <frequency>minutes(6)</frequency> <late-arrival cut-off="hours(4)"/> <clusters> <cluster name="primaryCluster" type="source"> <validity start="2016-05-23T00:00Z" end="2017-05-31T00:00Z"/> <retention limit="days(90)" action="delete"/> <import> <source name="mysql-db" tableName="employee"> <extract type="incremental"> <deltacolumn>modified_item</deltacolumn> <mergepolicy>append</mergepolicy> </extract> </source> <arguments> <argument name="--split-by" value="id"/> <argument name="--num-mappers" value="2"/> </arguments> </import> </cluster> </clusters> <locations> <location type="data" path="/falcon/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}"/> <location type="stats" path="/none"/> <location type="meta" path="/none"/> </locations> <notification type="email" to="xyz.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__xyz.com&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=0bHHCBjZnnogp83kfVwhfyxKkEWQizicw7CrGfPklGI&e=>"/> <ACL owner="hadoop" group="hadoop" permission="0755"/> <schema location="/none" provider="none"/> </feed> Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
