[
https://issues.apache.org/jira/browse/NIFI-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018025#comment-16018025
]
ASF GitHub Bot commented on NIFI-3942:
--------------------------------------
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/1831
@bbende I very much like the idea of this PR. I tried to test it out,
though, and it didn't behave as I was hoping.
I downloaded the GeoLite2 database for ASN and for City. I set the service
to lookup all fields. I then copied the City database to "db.mmdb" and
configured the lookup to load db.mmdb file. While running, I tried overwriting
db.mmdb with the ASN database so that we can see if it reloads properly.
Unfortunately, though, as soon as I overwrote the file, I got the following
stack trace:
```
2017-05-19 17:02:19,287 ERROR [Timer-Driven Process Thread-4]
o.a.n.processors.standard.LookupRecord
LookupRecord[id=17a48af1-015c-1000-29d3-2b7ebd7d4241] Failed to process
StandardFlowFileRecord[uuid=c25e503e-5f28-4b59-a05f-7462df2424b8,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1495227698674-1, container=default,
section=1], offset=36678, length=91],offset=0,name=1599597234048490,size=91]:
org.apache.nifi.processor.exception.ProcessException: Failed to lookup value
'216.58.195.228' in Lookup Service
org.apache.nifi.processor.exception.ProcessException: Failed to lookup
value '216.58.195.228' in Lookup Service
at
org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:218)
at
org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:58)
at
org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:136)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106)
at
org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:129)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.nifi.lookup.LookupFailureException: Failed to lookup
City information for IP Address /216.58.195.228
at
org.apache.nifi.lookup.maxmind.IPLookupService.lookup(IPLookupService.java:211)
at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy77.lookup(Unknown Source)
at
org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:216)
... 17 common frames omitted
Caused by: com.maxmind.db.InvalidDatabaseException: The MaxMind DB file's
search tree is corrupt: contains pointer larger than the database.
at com.maxmind.db.Reader.resolveDataPointer(Reader.java:196)
at com.maxmind.db.Reader.get(Reader.java:103)
at
org.apache.nifi.lookup.maxmind.DatabaseReader.get(DatabaseReader.java:150)
at
org.apache.nifi.lookup.maxmind.DatabaseReader.city(DatabaseReader.java:194)
at
org.apache.nifi.lookup.maxmind.IPLookupService.lookup(IPLookupService.java:209)
... 23 common frames omitted
```
If we are going to support reloading, then we need to support having the
file overwritten. I think we should wrap the entire lookup method in a
try/catch and catch InvalidDatabaseException. If this is caught, we should
reload the database and try again. If it fails a second time, then we should
just throw a LookupFailedException. Thoughts?
> IPLookupService should auto reload geo dictionary file
> ------------------------------------------------------
>
> Key: NIFI-3942
> URL: https://issues.apache.org/jira/browse/NIFI-3942
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.3.0
> Reporter: Joseph Witt
> Assignee: Bryan Bende
>
> Right now the IPLookupService introduced in NIFI-3863 loads the database
> onEnabled for the controller service. It needs to periodically check for a
> new database file and reload it automatically during calls to lookup.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
