Joe, looked at dmesg and see a few of these on one of the nodes...I wonder if it's an IO issue. I'll do some digging around
[5492401.012087] INFO: task java:15838 blocked for more than 120 seconds. .... .... .... On Wed, Jan 17, 2018 at 8:43 AM Joe Witt <[email protected]> wrote: > Well, I think i'm hoping it was the restart and not the change of > shard size because one I can understand and the other I cannot :) > > If you think you run into this again can you please get thread dumps > and share them? > > Thanks! > > On Wed, Jan 17, 2018 at 10:41 AM, dan young <[email protected]> wrote: > > Heya Joe, > > > > I'm not sure if this had anything to do with it, but I increased the > > nifi.provenance.repository.index.shard.size to 4GB from the default > 500MB, > > did a rolling restart of the nodes, and now the search seems to be > working. > > Not sure if it was that or the restart....but I'll keep an eye on it. > > > > Regards, > > > > Dano > > > > > > On Tue, Jan 16, 2018 at 7:57 PM Joe Witt <[email protected]> wrote: > >> > >> dan - i've seen behavior that frustrated me like this at times but it > >> was almost always based on me not realizing the timezone settings or > >> something else related to my query time versus the time on the system. > >> I believe you can activate more detailed logging for that class and > >> others to see information about the queries themselves. I'll try some > >> of this tomorrow and share more findings if able. > >> > >> If no luck then please be sure to file a JIRA. > >> > >> Thanks > >> > >> On Tue, Jan 16, 2018 at 7:26 PM, dan young <[email protected]> wrote: > >> > Hello, > >> > > >> > We're running a secure 3 node 1.4 cluster, and for some reason we're > not > >> > able to search processor provenance event. i.e. trying to search a > for > >> > a > >> > particular event by filename, but the results always come back empty, > >> > even > >> > though when I view the provenance event history I can see the event. > >> > Has > >> > anyone seen this, or can offer any suggestions on how to debug why I'm > >> > not > >> > able to search? > >> > > >> > I've tried shutting the cluster down and deleting all the > repositories a > >> > couple of times.... > >> > > >> > Here's my relevant config: > >> > > >> > > >> > > nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository > >> > > >> > nifi.provenance.repository.debug.frequency=1_000_000 > >> > > >> > nifi.provenance.repository.encryption.key.provider.implementation= > >> > > >> > nifi.provenance.repository.encryption.key.provider.location= > >> > > >> > nifi.provenance.repository.encryption.key.id= > >> > > >> > nifi.provenance.repository.encryption.key= > >> > > >> > > >> > # Persistent Provenance Repository Properties > >> > > >> > > >> > > nifi.provenance.repository.directory.default=/opt/nifi-common/provenance_repository > >> > > >> > nifi.provenance.repository.max.storage.time=24 hours > >> > > >> > nifi.provenance.repository.max.storage.size=5 GB > >> > > >> > nifi.provenance.repository.rollover.time=5 min > >> > > >> > nifi.provenance.repository.rollover.size=100 MB > >> > > >> > nifi.provenance.repository.query.threads=2 > >> > > >> > nifi.provenance.repository.index.threads=2 > >> > > >> > nifi.provenance.repository.compress.on.rollover=true > >> > > >> > nifi.provenance.repository.always.sync=false > >> > > >> > nifi.provenance.repository.journal.count=16 > >> > > >> > nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, > >> > Filename, > >> > ProcessorID, Relationship > >> > > >> > nifi.provenance.repository.indexed.attributes= > >> > > >> > nifi.provenance.repository.index.shard.size=500 MB >
