We use dockerized NiFi 1.6.0 in cluster mode (2 nodes) with external
zookeeper. LIstAzureBlobStorage processor is configured to run on primary
node. While new files appear on Azure Blob Storage, after some time
LIstAzureBlobStorage stops to generate flow files, cluster state for this
component constantly returns same value:
nifi_1 | 2018-04-18 15:18:13,333 DEBUG [Timer-Driven Process Thread-7]
o.a.n.p.a.storage.ListAzureBlobStorage
ListAzureBlobStorage[id=c9823e0d-b2af-161f-92d5-5860f1c39ea6] Returning
CLUSTER State: StandardStateMap[version=818,
values={processed.timestamp=1524062282000, listing.timestamp=1524062283000,
id.0=https://STORAGE_ACCOUNT_NAME.blob.core.windows.net/backup/20180418/xxx/Alarm/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json.gz,
id.1=https://STORAGE_ACCOUNT_NAME.blob.core.windows.net/backup/20180418/yyy/Alarm/yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy.json.gz}]
nifi_1 | 2018-04-18 15:18:13,333 DEBUG [Timer-Driven Process Thread-7]
o.a.n.p.a.storage.ListAzureBlobStorage
ListAzureBlobStorage[id=c9823e0d-b2af-161f-92d5-5860f1c39ea6] has chosen to
yield its resources; will not be scheduled to run again for 1000
milliseconds
nifi_1 | 2018-04-18 15:18:14,334 DEBUG [Timer-Driven Process Thread-2]
o.a.n.p.a.storage.ListAzureBlobStorage
ListAzureBlobStorage[id=c9823e0d-b2af-161f-92d5-5860f1c39ea6] Returning
CLUSTER State: StandardStateMap[version=818,
values={processed.timestamp=1524062282000, listing.timestamp=1524062283000,
id.0=https://STORAGE_ACCOUNT_NAME.blob.core.windows.net/backup/20180418/xxx/Alarm/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json.gz,
id.1=https://STORAGE_ACCOUNT_NAME.blob.core.windows.net/backup/20180418/yyy/Alarm/yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy.json.gz}]
nifi_1 | 2018-04-18 15:18:14,334 DEBUG [Timer-Driven Process Thread-2]
o.a.n.p.a.storage.ListAzureBlobStorage
ListAzureBlobStorage[id=c9823e0d-b2af-161f-92d5-5860f1c39ea6] has chosen to
yield its resources; will not be scheduled to run again for 1000
milliseconds
I thought it might be related to quit one of the node from the cluster and
starting of reelecting process, but I've tried to manually stop Primary Node
and everything went smoothly with reelection of new Primary and continuation
of getting new blob flow files even after returning of the stopped node back
to the cluster.
Let me know if you need additional details to help us resolve the issue.
--
Thanks,
Max
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
