I was away from Clojure for a year and I missed it. I am pleased to be
back. But I've forgotten certain common errors. I feel like this is
something I used to know but now I've lost the knowledge.
I'm using Factual's durable-queue to put a step inbetween the import of
large JSON files, and their writes to the database. This works fine on my
local MacBook Pro, but when I move to an EC2 instance, I'm instead getting
time-outs when durable-queue tries to read from the queue.
At start up the app creates a few of these workers, which run for as long
as the app is running:
(defn worker
[from-topics-to-persistence-queue current-database-connection]
(slingshot/try+
(loop [message (durable/take! from-topics-to-persistence-queue :message
60000 :timed-out!)]
(slingshot/try+
(log-seq " the message in the work function " message)
(when (= (type message) durable_queue.Task)
(advance message current-database-connection))
(catch Object o
(log-seq "error in worker function")
(durable/retry! message)
(log-seq o)))
(recur (durable/take! from-topics-to-persistence-queue :message 60000
:timed-out!)))
(catch Object o
(error o)
(slingshot/throw+ {
:type worker
:error o
:from-topics-to-persistence-queue
from-topics-to-persistence-queue
:current-database-connection
current-database-connection
}
))))
On the EC2 instance, I see in the output:
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO: the message in the work function
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO:
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO: :timed-out!
I'm using an environment var to set the path to the queue:
PATH_TO_DURABLE_QUEUE_S3_TO_DATABASE=/tmp/s3_to_database_queue
I feel like I ran into this problem 2 years ago, but I can't recall the
solution. Is the problem specific to AWS?
An important clue, I think, is that I'm not getting an error on the put!,
only on the take! Why would that be?
--- lawrence
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
