NoSuchMethodError when AOT'ing with Clojure 1.10

[Matthew Phillips]

Hi all,

I have an inexplicable runtime error with a leiningen-generated AOT uberjar 
that happens when using Clojure 1.10 that doesn't happen with 1.9 (I'm on 
Java 8 for both compilation and deployment).

The code in question looks like:

(defn make-apns-message [payload]
  (let [buffer (ByteBuffer/allocate 3072)]
(doto buffer
  (.put (byte 2))
  (.position (int 5))  ;; this fails
  (.put (byte 1))  

  ;;

  (.flip))
(Unpooled/wrappedBuffer buffer)))

When compiled with Clojure 1.10, the call to .position triggers this at 
runtime:

java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/
ByteBuffer;
at basis.apns$make_apns_message.invokeStatic(apns.clj:255)
at basis.apns$make_apns_message.invoke(apns.clj:249)
at basis.apns$send_queued_apns_messages.invokeStatic(apns.clj:162)
at basis.apns$send_queued_apns_messages.invoke(apns.clj:157)
at basis.apns$schedule_open_apns$fn__11654$fn__11655.invoke(apns.clj
:109)
at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2033)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Agent$Action.doRun(Agent.java:114)
at clojure.lang.Agent$Action.run(Agent.java:163)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


It's very strange because there is only one variant of position 
 
in a ByteBuffer that takes a parameter (and I'm compiling with 
*warn-on-reflection* set to catch this sort of thing). 

Any ideas?

Cheers,

Matt.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue when moving to Clojure 1.10

[Alex Miller]

On Tue, Jan 15, 2019 at 10:43 PM Didier  wrote:

> Okay, I found my issue, mostly.
>
> It seems that Clojure 1.9 and 1.10 both now have a
> clojure.core.specs.alpha.clj file.
>
> This file seems to be the only file that is not AOT compiled by the maven
> compile script for Clojure. Thus the Jar for Clojure only has it as source,
> and is missing the .classes for it.
>
> We have custom Ant tasks to compile Clojure, where we call compile-ns
> ourselves on our source namespaces.
>
> I was using Clojure 1.9 in our Ant tasks, which meant that as part of
> running compile-ns on our namespace, because they depend on Clojure core as
> well, clojure.core.specs.alpha.clj from the Clojure 1.9 jar was also being
> compiled.
>
> Thus, our project jar now contained the 1.9 classes for core specs.
>
> Now, we would use Clojure 1.10 to run our project, but the Clojure 1.9
> core specs fail to validate amp-impl of Clojure 1.10. I'm guessing it was
> changed along with its spec inside core specs.
>
> So we were getting a weird syntax error at runtime for every namespace
> that depend on clojure.core.
>
> I have a few questions based on this:
>
> 1) Why is clojure.core.specs.alpha.clj the only thing not AOT compiled in
> the Clojure jar?
>

clojure.core.specs.alpha is actually not part of Clojure proper, it's in
the dependencies. We don't compile the specs as we would then be bound to a
particular version of the spec.alpha dependency. We wanted it to be
possible to bump those versions but not have to be binary-compatible. (And
this is exactly the issue you're seeing - binary incompatibility of
compiled specs between versions.)

There are actually a few other namespaces in Clojure itself that are not
AOT compiled because they are conditionally included, or for other reasons.
Some of these special cases went away in Clojure 1.10 due to the bump in
minimum JDK which removed a lot of the special cases.


>
> 2) I realized that I probably shouldn't compile Clojure 1.10 code with
> Clojure 1.9, and I bumbed our Ant code to use Clojure 1.10, and that fixed
> it. So what are the compiler compatibility? Can newer Clojure versions
> compile older Clojure code? Is it only two version back? Is this tested? Or
> should I always use equal versions to compile? Is forward compatibility a
> thing? Etc.
>

I'm not sure you've accurately described the scenario here. In general, we
don't guarantee binary compatibility between versions of Clojure, however,
we do make every effort to maintain stability of the interfaces that
compiled Clojure classes rely on inside the language (all the Java core
like RT, etc). However, I don't think the problem here is with Clojure
itself, it's with spec.

What you're seeing are specs compiled with one version of spec.alpha, then
being run with a different version of spec.alpha. We have not been trying
to maintain binary compatibility for compiled specs (and have not been
compiling the core.specs to avoid this being an issue). Protocols tend to
be particularly problematic in this area (the spec interface is a
protocol), so that's a complicating factor here.


> 3) Is compile-ns transitive? I was surprised to see that it also compiles
> clojure.core.specs.alpha.
>

Yes. When compilation is on, it transitively compiles every namespace
loaded.

4) Isn't it a bit strange that if I AOT compile my code, my Jar gets .class
> files from Clojure core as well? Even though I don't get the error anymore,
> I still have clojure.core.specs.alpha classes from my jar on the classpath,
> and the source from Clojure on it as well.
>

Yes, it's one of the downsides of AOT. This has been discussed for a long
time (see https://dev.clojure.org/jira/browse/CLJ-322 for example). Having
both source and classes is totally fine - Clojure prefers the class file,
but there are issues (particularly with protocols) in having classes that
depend on source files.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue when moving to Clojure 1.10

[Didier]

Okay, I found my issue, mostly. 

It seems that Clojure 1.9 and 1.10 both now have a clojure.core.specs.alpha.clj 
file.

This file seems to be the only file that is not AOT compiled by the maven 
compile script for Clojure. Thus the Jar for Clojure only has it as source, and 
is missing the .classes for it.

We have custom Ant tasks to compile Clojure, where we call compile-ns ourselves 
on our source namespaces.

I was using Clojure 1.9 in our Ant tasks, which meant that as part of running 
compile-ns on our namespace, because they depend on Clojure core as well, 
clojure.core.specs.alpha.clj from the Clojure 1.9 jar was also being compiled.

Thus, our project jar now contained the 1.9 classes for core specs.

Now, we would use Clojure 1.10 to run our project, but the Clojure 1.9 core 
specs fail to validate amp-impl of Clojure 1.10. I'm guessing it was changed 
along with its spec inside core specs.

So we were getting a weird syntax error at runtime for every namespace that 
depend on clojure.core.

I have a few questions based on this:

1) Why is clojure.core.specs.alpha.clj the only thing not AOT compiled in the 
Clojure jar?

2) I realized that I probably shouldn't compile Clojure 1.10 code with Clojure 
1.9, and I bumbed our Ant code to use Clojure 1.10, and that fixed it. So what 
are the compiler compatibility? Can newer Clojure versions compile older 
Clojure code? Is it only two version back? Is this tested? Or should I always 
use equal versions to compile? Is forward compatibility a thing? Etc.

3) Is compile-ns transitive? I was surprised to see that it also compiles 
clojure.core.specs.alpha.

4) Isn't it a bit strange that if I AOT compile my code, my Jar gets .class 
files from Clojure core as well? Even though I don't get the error anymore, I 
still have clojure.core.specs.alpha classes from my jar on the classpath, and 
the source from Clojure on it as well.

Thanks

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue when moving to Clojure 1.10

[Alex Miller]

:import works with either (personally I prefer []).

> On Jan 15, 2019, at 7:00 PM, Matching Socks  wrote:
> 
> This might have nothing to do with it, but, doesn't :import use parens?
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "Clojure" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/clojure/qIA7GoAVtbE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue when moving to Clojure 1.10

[Matching Socks]

This might have nothing to do with it, but, doesn't :import use parens?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Bract: Data-driven Application initialisation framework

[Shantanu Kumar]

Hi,

I am happy to announce a data-driven application initialisation framework 
called Bract.

https://bract.github.io/

Bract provides a first class, declarative way to express application 
initialisation config and steps/workflow. Bract has modules for various 
purposes, and offers the necessary glue to bind an application. It can 
integrate with tests, REPL (Reloaded workflow) and CLI etc. There are 
example applications included to demonstrate how to use Bract. Gossamer is 
a minimal, extensible web framework built on top of Bract modules.

At SAP Concur, we have been using Bract for over a year in production to 
build REST(ish) microservices. I spoke about Bract at the IN/Clojure 2019 
conference recently, the slides for which are at the URL below:

https://speakerdeck.com/kumarshantanu/bract-a-minimal-dry-application-framework

I would be happy to receive any feedback, suggestions and questions about 
Bract. Please let me know what you think.


Shantanu

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

P.S. Actually it just a coincidence that in my use case I am not concerned 
with let say trees, but with "Records" in a "Table", which in contrast map 
somehow nicely to IPersistentMap (where I can put a lot of additional 
functionality in a transparent way) and a "Table" (where the sequence 
functions built in sequence functions actually desintegrate the "Table", 
which is undesired for me) which _could_ map to a "IPersistentVector", 
which makes me desire to have them as transparent to the rest of Clojure as 
possible and let them act as if the user would actually use vectors and 
maps and invoke additional functionality only when needed, while all the 
internal accounting/maintenance of the table of records is hidden from the 
user. I wanted that it is more like a vector of maps and not a Clojure 
sequence of maps, because the vector abstraction has some useful (not for 
databases in general, but but my specific use case) properties and because 
it is already extensible through the Clojure's collection protocols. On the 
other side (outside of Clojure) I could aim to maintain the other 
abstraction of lazy/chunked sequences which are produced by Clojure's 
sequence functions, but then again have to restitch them to "Table"-s and 
there is no protocol for it. On the third side one could think of a Table 
which always has at least some primary key - then one would rather use 
IPersistentMap for the Table itself, but this again doesn't provide a 
transparent solution for the sequence functions. And so on and so on. I 
just miss the glue which in all these cases, which would allow me to give 
to the end user a fully maintained and consistent "Table" without forcing 
him into wrapper functions or do-it-yourselfs instead of using the built 
ins - I wanted that for the end user only an implementation is different, 
but the whole interface remains pure Clojure :)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[greybird]

Thanks for explaining Plamen. Yes, it seems very difficult to treat a 
database abstraction as a regular Clojure map/vector. FWIW, in writing an 
app I think it works well to use SQL queries to return ordinary 
maps/vectors which can then be manipulated as usual. But I think you are 
doing something more ambitious. The interface to a database is something 
that is very tempting to try to improve, and I have tried several times 
without accomplishing very much in the end. :-)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

Hello Mark

the reason was that I want them to act as if they are maps and vectors for 
an end user developer and let him work with all the usual Clojure functions 
for these data structures, while internally they would have some different 
implementation and additional functionality (for unrelated to Clojure 
itself reasons). So, for me it is less about having less API objects, it 
was much more about transparency (e.g. HashMap and SortedMap have 
essentially the same interface and one just uses assoc/dissoc independent 
of the implementation except for the actual constructor). In my case this 
works well for the "Records" as they are the elements of the "Table" 
collection. But for the sequence functions operating on a "Table" - as Alex 
mentioned, they return own sequences of elements ("Records") actually and 
this with reason in Clojure's line of thought. And as both mentioned, this 
will need restitching at the end back into a "Table". I just hoped that I 
miss something and there would be a transparent way (e.g. through the 
Clojure collection/sequence interfaces) to do this restitching "(into 
(table) ..." without rerouting the user to custom wrappers or let him do it 
on his own. 

This will actually work painlessly, except that the user needs to use then 
the wrapper functions, where the mapping between a records and tables map 
more or less directly to maps and vectors. But if one would like to 
implement something different and more involved (e.g. 
column/sparse/off-heap store for the table or some indexing which is 
meaningful not for a single "Record", but for the "Table" as a whole) the a 
"Record" would be either only a view over specific indexes in the column 
arrays (in the naive case) or the "Records" would be some derived objects 
for use only by the end user, but physically detached from the actual 
storage - and in these more convoluted examples (or need) the transparency 
will get harder. So - it seems to me that to support both - element, 
collection and sequence operations, I'll have to go so or so through some 
own wrappers, which are probably less flexible than the built in functions, 
but keep the whole thing consistent together (and at the end will fuse the 
collection/sequence operations to maintain the "Table" abstraction).

Regards
Plamen

On Tuesday, January 15, 2019 at 6:43:09 PM UTC+1, greybird wrote:
>
> Hi Plamen, I don't have any advice to offer but I'm curious why you want 
> to bind the table and column type info directly onto the result set. If you 
> associate them in some other way, then you can just use plain maps and 
> vectors. Are you trying to have less total objects in your API?
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[greybird]

Hi Plamen, I don't have any advice to offer but I'm curious why you want to 
bind the table and column type info directly onto the result set. If you 
associate them in some other way, then you can just use plain maps and 
vectors. Are you trying to have less total objects in your API?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

Hello James

Yes. The (into (empty table) (comp (filter process?) (map process)) table) 
was exactly what I wanted for an end user to avoid to have to write, but 
just a (filter ... table), otherwise I could provide a wrapper like (defn 
myfilter [pred table] (into (empty table) (comp (filter process?) (map 
process)) table)), which I also wanted to avoid, but you and Alex are both 
right.

Thank you for taking time.

With best regards
Plamen

On Tuesday, January 15, 2019 at 3:47:13 PM UTC+1, James Reeves wrote:
>
> On Tue, 15 Jan 2019 at 13:58, > wrote:
>
>> So - do I miss something either in my knowledge or in the Clojure's 
>> documentation/implementation and is there a meaningful way to apply 
>> Clojure's and not mine filter/map/take/drop/sort etc. functions on a Table 
>> and to get a Table back, without going the monads/own functions for 
>> everything route or should I take it for granted that I can implement some 
>> custom types, but no way to get them be fully transparent to Clojure?
>>
>
> You could write:
>
> (into (empty coll) xform coll)
>
> Assuming coll is some implementation of IPersistentCollection and where 
> xform is your transducer.
>  
> So for example:
>
> (into (empty table) (comp (filter process?) (map process)) table)
>
> -- 
> James Reeves
> booleanknot.com
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

Hello Alex

On Tuesday, January 15, 2019 at 3:19:05 PM UTC+1, Alex Miller wrote:
>
> Well first, there is a conscious split between collection and sequence 
> functions. Collection functions take a collection and return the same 
> collection (and take the collection first) - things like conj, assoc, 
> dissoc, disj, etc. Those functions are all trait-based and "update" 
> operations are performed on the collection (via those trait interfaces) so 
> the collection has the opportunity to return a new variant of its own type.
>

Yes


> For sequences, these take and return sequences (really seqables), which is 
> a logical abstraction. Sequence functions inherently make new sequences and 
> you don't have an opportunity to influence how that happens (they mostly 
> build new lazy seqs from scratch). However, this is exactly the thing that 
> transducer operations do - they let you separate the operation from what 
> happens with the results. So using something `into`, with a transducer 
> chain, from your source collection to your target collection is again, only 
> using collection ops (conj) so you're back in the collection world.
>

Yes, this confirms my understanding and that there is not only no shortcut, 
but that it would be wrong to actually look for a shortcut given the way 
the abstractions are defined.
 

>
> Hopefully that confirms/answers what you're thinking through.
>
> Lot of thanks for taking time to reply!

With best regards
Plamen

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[James Reeves]

On Tue, 15 Jan 2019 at 13:58,  wrote:

> So - do I miss something either in my knowledge or in the Clojure's
> documentation/implementation and is there a meaningful way to apply
> Clojure's and not mine filter/map/take/drop/sort etc. functions on a Table
> and to get a Table back, without going the monads/own functions for
> everything route or should I take it for granted that I can implement some
> custom types, but no way to get them be fully transparent to Clojure?
>

You could write:

(into (empty coll) xform coll)

Assuming coll is some implementation of IPersistentCollection and where
xform is your transducer.

So for example:

(into (empty table) (comp (filter process?) (map process)) table)

-- 
James Reeves
booleanknot.com

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[Alex Miller]

Well first, there is a conscious split between collection and sequence 
functions. Collection functions take a collection and return the same 
collection (and take the collection first) - things like conj, assoc, 
dissoc, disj, etc. Those functions are all trait-based and "update" 
operations are performed on the collection (via those trait interfaces) so 
the collection has the opportunity to return a new variant of its own type.

For sequences, these take and return sequences (really seqables), which is 
a logical abstraction. Sequence functions inherently make new sequences and 
you don't have an opportunity to influence how that happens (they mostly 
build new lazy seqs from scratch). However, this is exactly the thing that 
transducer operations do - they let you separate the operation from what 
happens with the results. So using something `into`, with a transducer 
chain, from your source collection to your target collection is again, only 
using collection ops (conj) so you're back in the collection world.

Hopefully that confirms/answers what you're thinking through.


On Tuesday, January 15, 2019 at 7:58:07 AM UTC-6, plamen...@gmail.com wrote:
>
> Hello all
>
> while working on a daily basis where I use Clojure's native vectors/maps I 
> almost never experience the problem and even if - it is easy fixable with 
> something like (into [] ...), I have the following problem with custom data 
> structures and ask here for clarification if my understanding is correct.
>
> Imagine I try on the one side to represent something like a database table 
> in memory, while on the other to make it pluggable into all meaningful 
> sequence and vector/map functions in Clojure. In the most naive 
> implementation a table is a vector of maps. If i would like to add more 
> functionality, while trying to remain transparent to Clojure functions I 
> could implement a custom Record type (IPersistentMap etc.) for the rows 
> (for a custom storage, for column ordering, type checking etc.) and this 
> works. I could also implement a custom Table (IPersistentVector etc.) for 
> the actual collection of records while maintaining internally some indexes, 
> the schema of the table etc. The point is that if possible to apply let say 
> (filter.. or (take 2 ..., (sort .. incl. define/apply transducers. or 
> whatever other Clojure function compatible with these interfaces on the 
> Table/IPersistentVector/IPersistentCollection and get back a Table. 
>
> What I know:
> 1. Clojure's doc actually says that sequence functions return sequences 
> and say nothing about preserving the original type, the implementations of 
> the sequence functions do actually as advertised and return usually Cons-es.
> 2. monads, yes, but the point is not to force the user to use custom 
> functions instead of the built in ones.
> 3. IPersistentCollection has first/next/more methods which could return a 
> Table instead of a Record or a collection of them.
> 4. IPersistentVector has a cons method, but it is used by Clojure's conj 
> for example, but not inside the implementations of filter/take etc. which 
> create actual Cons objects
> 5. I could attach table like descriptions to each Record object (be it in 
> its metadata or else), but then enforcing that all Records share the same 
> Table data could get penalizing at runtime.
> 6. I know how Incanter and core.matrix try to solve some similar problems.
> 7. there are some vector functions which return actual vectors back (like 
> filterv, mapv), but also not the original collection type.
>
>
> So - do I miss something either in my knowledge or in the Clojure's 
> documentation/implementation and is there a meaningful way to apply 
> Clojure's and not mine filter/map/take/drop/sort etc. functions on a Table 
> and to get a Table back, without going the monads/own functions for 
> everything route or should I take it for granted that I can implement some 
> custom types, but no way to get them be fully transparent to Clojure?
>
> With best regards
> Plamen
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

P.S. This is a question about clarification/advice, not a critique to 
Clojure, as it actually works as advertised :)

On Tuesday, January 15, 2019 at 2:58:07 PM UTC+1, plamen...@gmail.com wrote:
>
> Hello all
>
> while working on a daily basis where I use Clojure's native vectors/maps I 
> almost never experience the problem and even if - it is easy fixable with 
> something like (into [] ...), I have the following problem with custom data 
> structures and ask here for clarification if my understanding is correct.
>
> Imagine I try on the one side to represent something like a database table 
> in memory, while on the other to make it pluggable into all meaningful 
> sequence and vector/map functions in Clojure. In the most naive 
> implementation a table is a vector of maps. If i would like to add more 
> functionality, while trying to remain transparent to Clojure functions I 
> could implement a custom Record type (IPersistentMap etc.) for the rows 
> (for a custom storage, for column ordering, type checking etc.) and this 
> works. I could also implement a custom Table (IPersistentVector etc.) for 
> the actual collection of records while maintaining internally some indexes, 
> the schema of the table etc. The point is that if possible to apply let say 
> (filter.. or (take 2 ..., (sort .. incl. define/apply transducers. or 
> whatever other Clojure function compatible with these interfaces on the 
> Table/IPersistentVector/IPersistentCollection and get back a Table. 
>
> What I know:
> 1. Clojure's doc actually says that sequence functions return sequences 
> and say nothing about preserving the original type, the implementations of 
> the sequence functions do actually as advertised and return usually Cons-es.
> 2. monads, yes, but the point is not to force the user to use custom 
> functions instead of the built in ones.
> 3. IPersistentCollection has first/next/more methods which could return a 
> Table instead of a Record or a collection of them.
> 4. IPersistentVector has a cons method, but it is used by Clojure's conj 
> for example, but not inside the implementations of filter/take etc. which 
> create actual Cons objects
> 5. I could attach table like descriptions to each Record object (be it in 
> its metadata or else), but then enforcing that all Records share the same 
> Table data could get penalizing at runtime.
> 6. I know how Incanter and core.matrix try to solve some similar problems.
> 7. there are some vector functions which return actual vectors back (like 
> filterv, mapv), but also not the original collection type.
>
>
> So - do I miss something either in my knowledge or in the Clojure's 
> documentation/implementation and is there a meaningful way to apply 
> Clojure's and not mine filter/map/take/drop/sort etc. functions on a Table 
> and to get a Table back, without going the monads/own functions for 
> everything route or should I take it for granted that I can implement some 
> custom types, but no way to get them be fully transparent to Clojure?
>
> With best regards
> Plamen
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Custom vectors/maps and sequence functions

[plamen . usenet]

Hello all

while working on a daily basis where I use Clojure's native vectors/maps I 
almost never experience the problem and even if - it is easy fixable with 
something like (into [] ...), I have the following problem with custom data 
structures and ask here for clarification if my understanding is correct.

Imagine I try on the one side to represent something like a database table 
in memory, while on the other to make it pluggable into all meaningful 
sequence and vector/map functions in Clojure. In the most naive 
implementation a table is a vector of maps. If i would like to add more 
functionality, while trying to remain transparent to Clojure functions I 
could implement a custom Record type (IPersistentMap etc.) for the rows 
(for a custom storage, for column ordering, type checking etc.) and this 
works. I could also implement a custom Table (IPersistentVector etc.) for 
the actual collection of records while maintaining internally some indexes, 
the schema of the table etc. The point is that if possible to apply let say 
(filter.. or (take 2 ..., (sort .. incl. define/apply transducers. or 
whatever other Clojure function compatible with these interfaces on the 
Table/IPersistentVector/IPersistentCollection and get back a Table. 

What I know:
1. Clojure's doc actually says that sequence functions return sequences and 
say nothing about preserving the original type, the implementations of the 
sequence functions do actually as advertised and return usually Cons-es.
2. monads, yes, but the point is not to force the user to use custom 
functions instead of the built in ones.
3. IPersistentCollection has first/next/more methods which could return a 
Table instead of a Record or a collection of them.
4. IPersistentVector has a cons method, but it is used by Clojure's conj 
for example, but not inside the implementations of filter/take etc. which 
create actual Cons objects
5. I could attach table like descriptions to each Record object (be it in 
its metadata or else), but then enforcing that all Records share the same 
Table data could get penalizing at runtime.
6. I know how Incanter and core.matrix try to solve some similar problems.
7. there are some vector functions which return actual vectors back (like 
filterv, mapv), but also not the original collection type.


So - do I miss something either in my knowledge or in the Clojure's 
documentation/implementation and is there a meaningful way to apply 
Clojure's and not mine filter/map/take/drop/sort etc. functions on a Table 
and to get a Table back, without going the monads/own functions for 
everything route or should I take it for granted that I can implement some 
custom types, but no way to get them be fully transparent to Clojure?

With best regards
Plamen

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.