Re: get blob meta exception

2024-01-18 Thread Devendar Rao
Hi Bipin, There are no disk space issues on the nimbus host. We also see this exception. I think this is the effect not the cause. Not completely sure though. 2024-01-18 11:47:16.018 o.a.s.t.ProcessFunction pool-29-thread-60 [ERROR] Internal error processing beginBlobDownload

get blob meta exception

2024-01-18 Thread Devendar Rao
Hi, We're constantly seeing issues in storm 2.3.0 with blobs with each topology deployment. Supervisor/nimbus dies after seeing the below exceptions. Is this a known issue? Are we hitting any blob cache size limits? Exceptions: "Could not download the blob with key:" o.a.s.d.n.Nimbus

Re: get blob meta exception

2024-01-18 Thread Bipin Prasad
Hello Devender,   The zookeeper entry is expected to be removed when the topology is killed. It appears that you are using local file store blob. So the blobs are (expected to be) on the nimbus. Can you check the mailbox directory and confirm whether or not the blob for this topology made it

Re: get blob meta exception

2024-01-18 Thread Devendar Rao
Thanks Bipin for the response. To add more details: Whenever a new topology is deployed nimbus doesn't respond and supervisor(s) go down. We have to restart the services to bring the cluster back to normal. Another error we see: o.a.s.b.BlobStoreUtils BLOB-STORE-TIMER [ERROR] Could not download

Re: get blob meta exception

2024-01-18 Thread Devendar Rao
Full stack trace: 2024-01-18 18:27:12.889 o.a.s.d.n.Nimbus pool-29-thread-22 [WARN] get blob meta exception. org.apache.storm.utils.WrappedKeyNotFoundException: topology-A-7-1704896101-stormjar.jar at org.apache.storm.blobstore. LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:258)

Re: get blob meta exception

2024-01-18 Thread Bipin Prasad
Can you post the full stack trace? I want to confirm that this is logged while trying to obtain the heartbeat.This message is a warning message and is not expected to shutdown nimbus. Sent from Yahoo Mail for iPhone On Thursday, January 18, 2024, 11:19 AM, Devendar Rao wrote: Hi, We're

get blob meta exception

2024-01-18 Thread Devendar Rao
Hi, We're constantly seeing issues in storm 2.3.0 with blobs with each topology deployment. Supervisor nodes die after seeing the below exceptions. Is this a known issue? Are we hitting any blob cache size limits? Exceptions: "Could not download the blob with key:" 2024-01-18 18:27:12.889

Docker images

2024-01-18 Thread Julien Nioche
Dear Storm community, You can now find version 2.6.0 of Apache Storm on Dockerhub , other versions have been deprecated. As you probably read in a separate thread, the code of the Docker images has been kindly donated by Elisey Zanko and is now at Apache. You can

Re: [PR] [STORM-4022] Do not log Exception when getting heartbeat timeout for dead topologies (storm)

2024-01-18 Thread via GitHub
jnioche merged PR #3617: URL: https://github.com/apache/storm/pull/3617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: