Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/1831
@bbende I very much like the idea of this PR. I tried to test it out,
though, and it didn't behave as I was hoping.
I downloaded the GeoLite2 database for ASN and for City. I set the service
to lookup all fields. I then copied the City database to "db.mmdb" and
configured the lookup to load db.mmdb file. While running, I tried overwriting
db.mmdb with the ASN database so that we can see if it reloads properly.
Unfortunately, though, as soon as I overwrote the file, I got the following
stack trace:
```
2017-05-19 17:02:19,287 ERROR [Timer-Driven Process Thread-4]
o.a.n.processors.standard.LookupRecord
LookupRecord[id=17a48af1-015c-1000-29d3-2b7ebd7d4241] Failed to process
StandardFlowFileRecord[uuid=c25e503e-5f28-4b59-a05f-7462df2424b8,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1495227698674-1, container=default,
section=1], offset=36678, length=91],offset=0,name=1599597234048490,size=91]:
org.apache.nifi.processor.exception.ProcessException: Failed to lookup value
'216.58.195.228' in Lookup Service
org.apache.nifi.processor.exception.ProcessException: Failed to lookup
value '216.58.195.228' in Lookup Service
at
org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:218)
at
org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:58)
at
org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:136)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106)
at
org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:129)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.nifi.lookup.LookupFailureException: Failed to lookup
City information for IP Address /216.58.195.228
at
org.apache.nifi.lookup.maxmind.IPLookupService.lookup(IPLookupService.java:211)
at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy77.lookup(Unknown Source)
at
org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:216)
... 17 common frames omitted
Caused by: com.maxmind.db.InvalidDatabaseException: The MaxMind DB file's
search tree is corrupt: contains pointer larger than the database.
at com.maxmind.db.Reader.resolveDataPointer(Reader.java:196)
at com.maxmind.db.Reader.get(Reader.java:103)
at
org.apache.nifi.lookup.maxmind.DatabaseReader.get(DatabaseReader.java:150)
at
org.apache.nifi.lookup.maxmind.DatabaseReader.city(DatabaseReader.java:194)
at
org.apache.nifi.lookup.maxmind.IPLookupService.lookup(IPLookupService.java:209)
... 23 common frames omitted
```
If we are going to support reloading, then we need to support having the
file overwritten. I think we should wrap the entire lookup method in a
try/catch and catch InvalidDatabaseException. If this is caught, we should
reload the database and try again. If it fails a second time, then we should
just throw a LookupFailedException. Thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
