[ANN] clojure.java.jdbc 0.7.0 Beta 4

[Sean Corfield]

[org.clojure/java.jdbc “0.7.0-beta4”] is available!

It fixes reflection warnings, it passes opts correctly from reducible-query to 
the underlying prepare-statement, and it completes the optional specs (compared 
to Beta 1), as well as correctly supporting the docstring contract of reduce.

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

From: Sean Corfield
Sent: Tuesday, July 4, 2017 7:27 PM
To: Clojure Mailing List
Subject: RE: [ANN] clojure.java.jdbc 0.7.0 Beta 1

> The required steps to setup streaming are different from database to
> database and I guess they may change with the driver version being used.

This is really the crux of the problem here – I’m not sure what java.jdbc can 
do generically to make this much easier.

Reviewing the code, you have found a bug! The options passed to reducible-query 
are passed to the reducible result set but they are not passed to 
db-query-with-result-set which is where the :fetch-size option should be passed 
down into prepare-statement. Ugh. I’ll fix that and put out a new release.

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

From: Ralf Schmitt
Sent: Tuesday, July 4, 2017 1:29 PM
To: Sean Corfield; Clojure Mailing 
List
Subject: Re: [ANN] clojure.java.jdbc 0.7.0 Beta 1

Sean Corfield  writes:

> Did you try this:
>
> (jdbc/with-db-connection [conn config/db]
>   (.setAutoCommit (jdbc/db-find-connection conn) false)
>   (into [] (take 2) (jdbc/reducible-query conn query {:fetch-size 500}))
>
> (I don’t have your sort of data set to test on with PostgreSQL)

Hi Sean,

that didn't work, i.e. I got an OutOfMemoryError. Even if it did, it
uses a lot of boilerplate code and looks rather 'pale' in comparison to

(into [] (take 2) (jdbc/reducible-query conn query {:use-streaming? true}))

It would also be nice from a callers perspective since he does not need
to know if the query should use streaming; it would be part of the
options passed to reducible-query.

I think it would make sense to add support for streaming into java.jdbc.

I'm not sure if this is feasible.

The required steps to setup streaming are different from database to
database and I guess they may change with the driver version being used.

I think you would need some functionality to let the user hook into the
process of setting up a connection for streaming. Though java.jdbc could
provide those hooks for some commonly used databases.

--
Cheers
Ralf


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [ANN] clojure.java.jdbc 0.7.0 Beta 1

[Sean Corfield]

> The required steps to setup streaming are different from database to
> database and I guess they may change with the driver version being used.

This is really the crux of the problem here – I’m not sure what java.jdbc can 
do generically to make this much easier.

Reviewing the code, you have found a bug! The options passed to reducible-query 
are passed to the reducible result set but they are not passed to 
db-query-with-result-set which is where the :fetch-size option should be passed 
down into prepare-statement. Ugh. I’ll fix that and put out a new release.

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

From: Ralf Schmitt
Sent: Tuesday, July 4, 2017 1:29 PM
To: Sean Corfield; Clojure Mailing List
Subject: Re: [ANN] clojure.java.jdbc 0.7.0 Beta 1

Sean Corfield  writes:

> Did you try this:
>
> (jdbc/with-db-connection [conn config/db]
>   (.setAutoCommit (jdbc/db-find-connection conn) false)
>   (into [] (take 2) (jdbc/reducible-query conn query {:fetch-size 500}))
>
> (I don’t have your sort of data set to test on with PostgreSQL)

Hi Sean,

that didn't work, i.e. I got an OutOfMemoryError. Even if it did, it
uses a lot of boilerplate code and looks rather 'pale' in comparison to

(into [] (take 2) (jdbc/reducible-query conn query {:use-streaming? true}))

It would also be nice from a callers perspective since he does not need
to know if the query should use streaming; it would be part of the
options passed to reducible-query.

I think it would make sense to add support for streaming into java.jdbc.

I'm not sure if this is feasible.

The required steps to setup streaming are different from database to
database and I guess they may change with the driver version being used.

I think you would need some functionality to let the user hook into the
process of setting up a connection for streaming. Though java.jdbc could
provide those hooks for some commonly used databases.

-- 
Cheers
Ralf

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] clojure.java.jdbc 0.7.0 Beta 1

[Ralf Schmitt]

Sean Corfield  writes:

> Did you try this:
>
> (jdbc/with-db-connection [conn config/db]
>   (.setAutoCommit (jdbc/db-find-connection conn) false)
>   (into [] (take 2) (jdbc/reducible-query conn query {:fetch-size 500}))
>
> (I don’t have your sort of data set to test on with PostgreSQL)

Hi Sean,

that didn't work, i.e. I got an OutOfMemoryError. Even if it did, it
uses a lot of boilerplate code and looks rather 'pale' in comparison to

(into [] (take 2) (jdbc/reducible-query conn query {:use-streaming? true}))

It would also be nice from a callers perspective since he does not need
to know if the query should use streaming; it would be part of the
options passed to reducible-query.

I think it would make sense to add support for streaming into java.jdbc.

I'm not sure if this is feasible.

The required steps to setup streaming are different from database to
database and I guess they may change with the driver version being used.

I think you would need some functionality to let the user hook into the
process of setting up a connection for streaming. Though java.jdbc could
provide those hooks for some commonly used databases.

-- 
Cheers
Ralf

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [ANN] clojure.java.jdbc 0.7.0 Beta 1

[Sean Corfield]

Did you try this:

(jdbc/with-db-connection [conn config/db]
  (.setAutoCommit (jdbc/db-find-connection conn) false)
  (into [] (take 2) (jdbc/reducible-query conn query {:fetch-size 500}))

(I don’t have your sort of data set to test on with PostgreSQL)

Sent from Mail for Windows 10

From: Ralf Schmitt
Sent: Tuesday, July 4, 2017 5:54 AM
To: Clojure Mailing List; Sean Corfield
Cc: Java.JDBC Mailing List
Subject: Re: [ANN] clojure.java.jdbc 0.7.0 Beta 1

Sean Corfield  writes:

> A new function, reducible-query, has been added that accepts a db-spec
> and a SQL/parameters vector, and returns a “reducible collection”. The
> query doesn’t actually run until you reduce it. At that point, it sets
> up the PreparedStatement, runs the query, and processes the ResultSet
> – by creating an interim “reducible collection” version of that result
> set and delegating your reduce operation to that. The connection is
> automatically closed when reduction completes, either by processing
> the entire result set or by returning a ‘reduced’ value.

How does one force the jdbc driver to return results lazily when using
reducible-query?

Currently I'm using the following code to force postgresql into
streaming mode:

(jdbc/with-db-connection [conn config/db]
  (.setAutoCommit (jdbc/db-find-connection conn) false)
  (jdbc/query conn
  query
  {:fetch-size 500
   :result-set-fn (fn [coll] (doall (take 2 coll)))}))

(config/db is map describing a postgresql connection, query is an sql
query)

This works as expected.

Trying to to the same with reducible-query with something like:

(into [] (take 2) (jdbc/reducible-query config/db query {:fetch-size 500}))

results in a 'GC overhead limit exceeded' error. The postgresql driver
isn't streaming the results, but instead returns the whole result in one
large response.

I don't have the ability to call .setAutoCommit on the connection in
that case. 

I would very much like an easy way to turn on streaming (e.g. by passing
{:use-streaming? true} as an option).

Do you have any plans for that?

Thanks for your work on java.jdbc.

-- 
Cheers
Ralf

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Simultaneous development of a clojure project and an internal util library

[Stuart Sierra]

Hi Istvan,

I often work with multiple source directories by temporarily adding them to 
the :source-paths 

 in 
a Leiningen profile 
.

If you have a mono-repo or a consistent pattern for checking out multiple 
repos, you can use relative paths in :source-paths.

–S

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: topic-fn returning more than one topic

[Stuart Sierra]

Hi Oded,

I think you would have to implement your own message-routing to achieve 
this. Create one "output" mult for each topic, subscribe to them with 
`tap`, and route the messages to the channels as they arrive.

Pseudo-code:
(go-loop []
  (when-some [message (! topic-ch message)))

This is quite similar to the go-loop in the implementation of `pub`: async.clj 
line 886 

.

–S

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Spec's not being checked on macro expansion

[Stuart Sierra]

Hi Ed,

I could not reproduce the behavior you described with Leiningen. Here is my 
test script:
https://gist.github.com/stuartsierra/ca2b9bd7072224e099beb8b95b49b69c

This issue might be caused by some pre-compiled files on the classpath when 
you run `lein test`. Since macros are evaluated during compilation, if the 
namespace using the macro was AOT-compiled then the macro would not be 
evaluated again. Try running `lein clean` or the Gradle equivalent, and 
make sure there are no stale AOT-compiled .class files in the /target 
directory.

–S

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] clojure.java.jdbc 0.7.0 Beta 1

[Ralf Schmitt]

Sean Corfield  writes:

> A new function, reducible-query, has been added that accepts a db-spec
> and a SQL/parameters vector, and returns a “reducible collection”. The
> query doesn’t actually run until you reduce it. At that point, it sets
> up the PreparedStatement, runs the query, and processes the ResultSet
> – by creating an interim “reducible collection” version of that result
> set and delegating your reduce operation to that. The connection is
> automatically closed when reduction completes, either by processing
> the entire result set or by returning a ‘reduced’ value.

How does one force the jdbc driver to return results lazily when using
reducible-query?

Currently I'm using the following code to force postgresql into
streaming mode:

(jdbc/with-db-connection [conn config/db]
  (.setAutoCommit (jdbc/db-find-connection conn) false)
  (jdbc/query conn
  query
  {:fetch-size 500
   :result-set-fn (fn [coll] (doall (take 2 coll)))}))

(config/db is map describing a postgresql connection, query is an sql
query)

This works as expected.

Trying to to the same with reducible-query with something like:

(into [] (take 2) (jdbc/reducible-query config/db query {:fetch-size 500}))

results in a 'GC overhead limit exceeded' error. The postgresql driver
isn't streaming the results, but instead returns the whole result in one
large response.

I don't have the ability to call .setAutoCommit on the connection in
that case. 

I would very much like an easy way to turn on streaming (e.g. by passing
{:use-streaming? true} as an option).

Do you have any plans for that?

Thanks for your work on java.jdbc.

-- 
Cheers
Ralf

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async/close! locks on chans with pending transforms

[Vitalie Spinu]

>> On Mon, Jul 03 2017 21:18, Timothy Baldridge wrote:

> This means, if you want to execute take correctly you must ensure that only
> one thread executes the take instance at one time, since channels already
> operate via a channel-level lock, it makes sense to run the transducer
> inside the channels' lock.

I looked at the code and got somewhat surprised that transducer can be
potentially called within all 3 methods - take!, put! and close!. Not sure if
this could be done differently, but given this implementation it indeed makes
sense to block all 3 methods.

I think the docs regarding the blocking behavior and the timing of transducer's
execution could be improved. I always naively assumed that transducers are
executed on the same thread within put!, but in fact xform is called only when
the value is added to the buffer and can happen both on current thread and on
the thread pool.

This is what I dug so far regarding put!:

1) (half-full buf and pending takes)
 -> transform and add to buf on the same thread

2) (nil buf or full buf) and pending takes
 -> dispatch to pending take on the thread pool
   
3) (half full buf and no pending takes)
 -> transform and add to buf on the same thread
 
4) (nil of full buf)
 -> append to puts list


I find 2 troublesome. It means that this line:

  
https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/channels.clj#L135

can be reached in the corner case when buffer is full and there are pending
takes. If so then the current val is dispatched to the leading pending take
bypassing buf completely. I think that line is intended to be reached only for
unbuffered chans. Am I mistaken here?


  Vitalie

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.