Sean, thanks for asking. The function is so basic that I didn't include it,
but here it is:
(defn fetch [mysql-db]
(slingshot/try+
(jdbc/query mysql-db
[" SELECT p.id as profile_id, p.name as profile_name,
p.headquarters_addr1, p.headquarters_city, p.headquarters_state_code,
headquarters_country_code, cw.url FROM company_profile p LEFT JOIN
company_website cw ON p.id = cw.company_profile_id WHERE p.name is not null
and p.name != '' "])
(catch Object o
(errors/error o "" " query to database had a problem "))))
On Wednesday, July 12, 2017 at 2:09:32 AM UTC-4, Sean Corfield wrote:
>
> What is the ‘fetch’ function here?
>
>
>
> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
> An Architect's View -- http://corfield.org/
>
> "If you're not annoying somebody, you're not really alive."
> -- Margaret Atwood
>
>
>
> *From: *lawrence...@gmail.com <javascript:>
> *Sent: *Tuesday, July 11, 2017 1:18 PM
> *To: *Clojure <javascript:>
> *Subject: *Re: I can only get the first item of a lazyseq via a Manifold
> stream, and I can't get/find an Exception
>
>
>
> Justin, thanks. I'd like to ask a follow up question. To be clear, if I go
> into (doseq) then this works fine:
>
>
>
> (defn start []
>
> (let [
>
> config (get-config)
>
> mysql-db {
>
> :dbtype "mysql"
>
> :dbname (get-in config [:database :config :connection
> :database])
>
> :user (get-in config [:database :config :connection
> :user] )
>
> :password (get-in config [:database :config :connection
> :password] )
>
> :host (get-in config [:database :config :connection
> :host])
>
> }
>
> data (fetch mysql-db)
>
> ]
>
> (slingshot/try+
>
> (doseq [row data]
>
> (queue/enqueue row))
>
> (catch Object o
>
> (errors/error o "" "error in query_database/start: ")))))
>
>
>
>
>
> but if I do this instead:
>
>
>
> ;;;;;;; (doseq [row data]
>
> (queue/enqueue data))
>
>
>
> and then enqueue does this:
>
>
>
> (->> (ms/->source data)
>
> (ms/onto executor)
>
> (ms/map api/query))
>
>
>
>
>
> The first row comes through fine, but then the second row is never
> processed.
>
>
>
> So are you suggesting that simply passing "data" from one function to the
> next is enough to lose the database context? But only after the first row
> has been pulled?
>
>
>
>
>
>
>
>
>
>
>
>
> On Tuesday, July 11, 2017 at 1:44:15 PM UTC-4, Justin Smith wrote:
>
> My first suspicion would be that by the time you access the second
> element, you have exited the context of your database transaction, so
> there's no data stream available to get it from. Lazyness doesn't tend to
> mix well with stateful resources and contexts.
>
>
>
> On Mon, Jul 10, 2017 at 9:45 PM <lawrence...@gmail.com> wrote:
>
> Okay, that was a deadend. After going through line by line, I could pretty
> well rule out any kind of Exception or Error. The first row from the
> database seems to go through all of the functions perfectly, without any
> errors. But the second row never gets called.
>
>
>
> Which takes me back to where I started. Why can't I give a lazy-seq to a
> Manifold Stream and simply have a function map over the stream?
>
>
>
> On Monday, July 10, 2017 at 11:25:43 PM UTC-4, lawrence...@gmail.com
> wrote:
>
>
>
> Once again, my lack of knowledge of Java trips me up. Manifold relies on
> Dirigiste, which relies on Java's Executor Service. I see a bit here:
>
>
>
> http://www.nurkiewicz.com/2014/11/executorservice-10-tips-and-tricks.html
>
>
>
> Nurkiewicz writes:
>
> "I got bitten by that too many times: it won't print *anything*. No sign
> of java.lang.ArithmeticException: / by zero, nothing. Thread pool just
> swallows this exception, as if it never happened. If it was a good'ol
> java.lang.Thread created from scratch, UncaughtExceptionHandler
> <https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.UncaughtExceptionHandler.html>
> could
> work. "
>
>
>
> I suspect I'm facing something like that.
>
>
>
>
>
> On Monday, July 10, 2017 at 8:28:03 PM UTC-4, lawrence...@gmail.com
> wrote:
>
> By the way, this code works fine if I go into a (doseq) a level above
> enqueue, and then put each individual row onto the stream. Then the code
> loops over all of the rows. But that seems to defeat the whole point of
> using something like Manifold. I want to be able to put the whole lazy-seq
> on the stream. That is supposed to work, yes?
>
>
>
> On Monday, July 10, 2017 at 8:18:07 PM UTC-4, lawrence...@gmail.com
> wrote:
>
> I'm using Zach Tellman's excellent Manifold library, though I admit I
> don't fully understand it.
>
>
>
> My code queries a MySQL database and then needs to do some processing on
> each row retrieved. I copy-and-pasted some code from the documentation for
> Manifold:
>
>
>
>
>
> ;; 2017-07-10 -- we want a thread pool. I'm arbitrarily choosing 200
> threads.
>
> ;; query_database/start will pull data from the database and dump it onto a
>
> ;; stream below, at which point each row should be assigned to one of the
> rows
>
> ;; on our thread pool.
>
> (def executor (me/fixed-thread-executor 200))
>
>
>
>
>
> (defn enqueue
>
> [sequence-from-database]
>
> (slingshot/try+
>
> (println "the type of the object from the database: " (type
> sequence-from-database))
>
> (->> (ms/->source sequence-from-database)
>
> (ms/onto executor)
>
> (ms/map api/query))
>
> (catch Object o
>
> (println " message queue is not happy about the message we were
> given")
>
> (errors/error o "" " we tried to put something on the message queue,
> but we got an error "))))
>
>
>
> The line where I print out the type assures that I'm dealing with a
> LazySeq.
>
>
>
> The code prints out the type and the first row:
>
>
>
>
>
> the type of the object from the database: clojure.lang.LazySeq
>
>
>
> in query_api/query {:profile_id 2, :profile_name Mike Shaw Automotive
> Group, :headquarters_addr1 90 Madison St., :headquarters_city Denver,
> :headquarters_state_code CO, :headquarters_country_code US, :url
> mikeshawauto.com}
>
>
>
> and then it stops. I assume there must be an Exception or Error happening,
> but I can't find it. I've added as many general Catch clauses as I could:
>
>
>
>
>
> (defn query
>
> [row & args]
>
>
>
> (println " in query_api/query " row)
>
>
>
> (let [config (if args
>
> (first args))
>
> ]
>
> ;; 2017-03-30 -- the API is overwhelmed and all I get is Socket
> Timeout errors
>
> (Thread/sleep 300)
>
>
>
> (slingshot/try+
>
> (call-api row)
>
> (catch Object o
>
> (println " error : " o)
>
>
>
> ;;(errors/error o row " we tried to call-api, but we got this error
> ")
>
>
>
>
>
> )
>
>
>
> (catch Error e
>
> (println " there Error: " e))
>
> )))
>
>
>
> So if I do:
>
>
>
> java -jar scan-database-find-similar-items-standalone.jar
>
>
>
>
>
> The code runs till it prints out the first row from the database, and then
> it stops. Nothing else happens. There are no error messages.
>
>
>
> What did I miss?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com <javascript:>
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com <javascript:>
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+u...@googlegroups.com <javascript:>.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
