Re: Custom vectors/maps and sequence functions

[plamen . usenet]

P.S. Actually it just a coincidence that in my use case I am not concerned 
with let say trees, but with "Records" in a "Table", which in contrast map 
somehow nicely to IPersistentMap (where I can put a lot of additional 
functionality in a transparent way) and a "Table" (where the sequence 
functions built in sequence functions actually desintegrate the "Table", 
which is undesired for me) which _could_ map to a "IPersistentVector", 
which makes me desire to have them as transparent to the rest of Clojure as 
possible and let them act as if the user would actually use vectors and 
maps and invoke additional functionality only when needed, while all the 
internal accounting/maintenance of the table of records is hidden from the 
user. I wanted that it is more like a vector of maps and not a Clojure 
sequence of maps, because the vector abstraction has some useful (not for 
databases in general, but but my specific use case) properties and because 
it is already extensible through the Clojure's collection protocols. On the 
other side (outside of Clojure) I could aim to maintain the other 
abstraction of lazy/chunked sequences which are produced by Clojure's 
sequence functions, but then again have to restitch them to "Table"-s and 
there is no protocol for it. On the third side one could think of a Table 
which always has at least some primary key - then one would rather use 
IPersistentMap for the Table itself, but this again doesn't provide a 
transparent solution for the sequence functions. And so on and so on. I 
just miss the glue which in all these cases, which would allow me to give 
to the end user a fully maintained and consistent "Table" without forcing 
him into wrapper functions or do-it-yourselfs instead of using the built 
ins - I wanted that for the end user only an implementation is different, 
but the whole interface remains pure Clojure :)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

Hello Mark

the reason was that I want them to act as if they are maps and vectors for 
an end user developer and let him work with all the usual Clojure functions 
for these data structures, while internally they would have some different 
implementation and additional functionality (for unrelated to Clojure 
itself reasons). So, for me it is less about having less API objects, it 
was much more about transparency (e.g. HashMap and SortedMap have 
essentially the same interface and one just uses assoc/dissoc independent 
of the implementation except for the actual constructor). In my case this 
works well for the "Records" as they are the elements of the "Table" 
collection. But for the sequence functions operating on a "Table" - as Alex 
mentioned, they return own sequences of elements ("Records") actually and 
this with reason in Clojure's line of thought. And as both mentioned, this 
will need restitching at the end back into a "Table". I just hoped that I 
miss something and there would be a transparent way (e.g. through the 
Clojure collection/sequence interfaces) to do this restitching "(into 
(table) ..." without rerouting the user to custom wrappers or let him do it 
on his own. 

This will actually work painlessly, except that the user needs to use then 
the wrapper functions, where the mapping between a records and tables map 
more or less directly to maps and vectors. But if one would like to 
implement something different and more involved (e.g. 
column/sparse/off-heap store for the table or some indexing which is 
meaningful not for a single "Record", but for the "Table" as a whole) the a 
"Record" would be either only a view over specific indexes in the column 
arrays (in the naive case) or the "Records" would be some derived objects 
for use only by the end user, but physically detached from the actual 
storage - and in these more convoluted examples (or need) the transparency 
will get harder. So - it seems to me that to support both - element, 
collection and sequence operations, I'll have to go so or so through some 
own wrappers, which are probably less flexible than the built in functions, 
but keep the whole thing consistent together (and at the end will fuse the 
collection/sequence operations to maintain the "Table" abstraction).

Regards
Plamen

On Tuesday, January 15, 2019 at 6:43:09 PM UTC+1, greybird wrote:
>
> Hi Plamen, I don't have any advice to offer but I'm curious why you want 
> to bind the table and column type info directly onto the result set. If you 
> associate them in some other way, then you can just use plain maps and 
> vectors. Are you trying to have less total objects in your API?
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

Hello James

Yes. The (into (empty table) (comp (filter process?) (map process)) table) 
was exactly what I wanted for an end user to avoid to have to write, but 
just a (filter ... table), otherwise I could provide a wrapper like (defn 
myfilter [pred table] (into (empty table) (comp (filter process?) (map 
process)) table)), which I also wanted to avoid, but you and Alex are both 
right.

Thank you for taking time.

With best regards
Plamen

On Tuesday, January 15, 2019 at 3:47:13 PM UTC+1, James Reeves wrote:
>
> On Tue, 15 Jan 2019 at 13:58, > wrote:
>
>> So - do I miss something either in my knowledge or in the Clojure's 
>> documentation/implementation and is there a meaningful way to apply 
>> Clojure's and not mine filter/map/take/drop/sort etc. functions on a Table 
>> and to get a Table back, without going the monads/own functions for 
>> everything route or should I take it for granted that I can implement some 
>> custom types, but no way to get them be fully transparent to Clojure?
>>
>
> You could write:
>
> (into (empty coll) xform coll)
>
> Assuming coll is some implementation of IPersistentCollection and where 
> xform is your transducer.
>  
> So for example:
>
> (into (empty table) (comp (filter process?) (map process)) table)
>
> -- 
> James Reeves
> booleanknot.com
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

Hello Alex

On Tuesday, January 15, 2019 at 3:19:05 PM UTC+1, Alex Miller wrote:
>
> Well first, there is a conscious split between collection and sequence 
> functions. Collection functions take a collection and return the same 
> collection (and take the collection first) - things like conj, assoc, 
> dissoc, disj, etc. Those functions are all trait-based and "update" 
> operations are performed on the collection (via those trait interfaces) so 
> the collection has the opportunity to return a new variant of its own type.
>

Yes


> For sequences, these take and return sequences (really seqables), which is 
> a logical abstraction. Sequence functions inherently make new sequences and 
> you don't have an opportunity to influence how that happens (they mostly 
> build new lazy seqs from scratch). However, this is exactly the thing that 
> transducer operations do - they let you separate the operation from what 
> happens with the results. So using something `into`, with a transducer 
> chain, from your source collection to your target collection is again, only 
> using collection ops (conj) so you're back in the collection world.
>

Yes, this confirms my understanding and that there is not only no shortcut, 
but that it would be wrong to actually look for a shortcut given the way 
the abstractions are defined.
 

>
> Hopefully that confirms/answers what you're thinking through.
>
> Lot of thanks for taking time to reply!

With best regards
Plamen

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Custom vectors/maps and sequence functions

[plamen . usenet]

P.S. This is a question about clarification/advice, not a critique to 
Clojure, as it actually works as advertised :)

On Tuesday, January 15, 2019 at 2:58:07 PM UTC+1, plamen...@gmail.com wrote:
>
> Hello all
>
> while working on a daily basis where I use Clojure's native vectors/maps I 
> almost never experience the problem and even if - it is easy fixable with 
> something like (into [] ...), I have the following problem with custom data 
> structures and ask here for clarification if my understanding is correct.
>
> Imagine I try on the one side to represent something like a database table 
> in memory, while on the other to make it pluggable into all meaningful 
> sequence and vector/map functions in Clojure. In the most naive 
> implementation a table is a vector of maps. If i would like to add more 
> functionality, while trying to remain transparent to Clojure functions I 
> could implement a custom Record type (IPersistentMap etc.) for the rows 
> (for a custom storage, for column ordering, type checking etc.) and this 
> works. I could also implement a custom Table (IPersistentVector etc.) for 
> the actual collection of records while maintaining internally some indexes, 
> the schema of the table etc. The point is that if possible to apply let say 
> (filter.. or (take 2 ..., (sort .. incl. define/apply transducers. or 
> whatever other Clojure function compatible with these interfaces on the 
> Table/IPersistentVector/IPersistentCollection and get back a Table. 
>
> What I know:
> 1. Clojure's doc actually says that sequence functions return sequences 
> and say nothing about preserving the original type, the implementations of 
> the sequence functions do actually as advertised and return usually Cons-es.
> 2. monads, yes, but the point is not to force the user to use custom 
> functions instead of the built in ones.
> 3. IPersistentCollection has first/next/more methods which could return a 
> Table instead of a Record or a collection of them.
> 4. IPersistentVector has a cons method, but it is used by Clojure's conj 
> for example, but not inside the implementations of filter/take etc. which 
> create actual Cons objects
> 5. I could attach table like descriptions to each Record object (be it in 
> its metadata or else), but then enforcing that all Records share the same 
> Table data could get penalizing at runtime.
> 6. I know how Incanter and core.matrix try to solve some similar problems.
> 7. there are some vector functions which return actual vectors back (like 
> filterv, mapv), but also not the original collection type.
>
>
> So - do I miss something either in my knowledge or in the Clojure's 
> documentation/implementation and is there a meaningful way to apply 
> Clojure's and not mine filter/map/take/drop/sort etc. functions on a Table 
> and to get a Table back, without going the monads/own functions for 
> everything route or should I take it for granted that I can implement some 
> custom types, but no way to get them be fully transparent to Clojure?
>
> With best regards
> Plamen
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Custom vectors/maps and sequence functions

[plamen . usenet]

Hello all

while working on a daily basis where I use Clojure's native vectors/maps I 
almost never experience the problem and even if - it is easy fixable with 
something like (into [] ...), I have the following problem with custom data 
structures and ask here for clarification if my understanding is correct.

Imagine I try on the one side to represent something like a database table 
in memory, while on the other to make it pluggable into all meaningful 
sequence and vector/map functions in Clojure. In the most naive 
implementation a table is a vector of maps. If i would like to add more 
functionality, while trying to remain transparent to Clojure functions I 
could implement a custom Record type (IPersistentMap etc.) for the rows 
(for a custom storage, for column ordering, type checking etc.) and this 
works. I could also implement a custom Table (IPersistentVector etc.) for 
the actual collection of records while maintaining internally some indexes, 
the schema of the table etc. The point is that if possible to apply let say 
(filter.. or (take 2 ..., (sort .. incl. define/apply transducers. or 
whatever other Clojure function compatible with these interfaces on the 
Table/IPersistentVector/IPersistentCollection and get back a Table. 

What I know:
1. Clojure's doc actually says that sequence functions return sequences and 
say nothing about preserving the original type, the implementations of the 
sequence functions do actually as advertised and return usually Cons-es.
2. monads, yes, but the point is not to force the user to use custom 
functions instead of the built in ones.
3. IPersistentCollection has first/next/more methods which could return a 
Table instead of a Record or a collection of them.
4. IPersistentVector has a cons method, but it is used by Clojure's conj 
for example, but not inside the implementations of filter/take etc. which 
create actual Cons objects
5. I could attach table like descriptions to each Record object (be it in 
its metadata or else), but then enforcing that all Records share the same 
Table data could get penalizing at runtime.
6. I know how Incanter and core.matrix try to solve some similar problems.
7. there are some vector functions which return actual vectors back (like 
filterv, mapv), but also not the original collection type.


So - do I miss something either in my knowledge or in the Clojure's 
documentation/implementation and is there a meaningful way to apply 
Clojure's and not mine filter/map/take/drop/sort etc. functions on a Table 
and to get a Table back, without going the monads/own functions for 
everything route or should I take it for granted that I can implement some 
custom types, but no way to get them be fully transparent to Clojure?

With best regards
Plamen

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: clojure.spec, conform and returned value

[plamen . usenet]

Amazing! 

Lot of thanks Josh! This solves everything. It is even written in the API 
doc, but I missed the important piece of "(possibly converted) value"...

With best regards
Plamen

On Tuesday, October 11, 2016 at 8:49:16 PM UTC+2, Josh Tilles wrote:
>
> I think you’re looking for conformer 
> ;
>  
> the spec it produces will pass along the converted value instead of the 
> original input. It could be used like:
> (defn str->double [s]
>   (try
> (Double/valueOf s)
> (catch NumberFormatException _
>   :clojure.spec/invalid)))
>
> (s/def ::transforming-check (s/and string? (s/conformer str->double) ::
> bigger-than-zero?))
>
> I hope that helps with your original problem, but I’ll defer to the more 
> experienced folks here for your questions about the spirit/aim of 
> clojure.spec.
>
> On Tuesday, October 11, 2016 at 2:30:18 PM UTC-4, plamen...@gmail.com 
> wrote:
>>
>> Hello,
>>
>> I have a problem which is probably not in the spirit of clojure.spec as 
>> being a library for "only" checking/generating valid values, but of 
>> substantial practical value for my use case:
>>
>> Let say I have a function for checking if a double precision number is 
>> parsable from a string (where possible performance penalties because of 
>> communicating non-parsable strings through try/catch in the implementation 
>> is not part of my question):
>>
>>
>> (defn str-parsable-double? [s]
>>   "Note the IMPLIED transformation from string to a double"
>>   (try
>> (Double/valueOf s)
>> (catch NumberFormatException e
>>   false)))
>>
>>
>> and have a spec using this function:
>>
>> (s/def ::str->double str-parsable-double?)
>>
>>
>> We could actually chain checks for a single value as in:
>>
>> (s/def ::str->double (s/and string? str-parsable-double?))
>>
>> and we could check for the conformity of a value as per:
>>
>> (s/conform ::str->double "3.14")
>> => "3.14"
>> and 
>> (s/conform ::str->double "Pizza")
>> => :clojure.spec/invalid
>>
>> Until now everything is ok, now my the questions:
>>
>> In my real world example I need a chain of a combination of checks and 
>> transformations in the following way:
>> ::check1 ::check2 ::check-and-transform3 ::check-4-on-transformed-value 
>> ::check-5-on-transformed value
>> or something like
>> (s/def ::transforming-check (s/and string? ::str->double ::bigger-than-zero? 
>> ::smaller-than-pi ... etc ...))
>>
>> As the result of 
>> (s/conform ::str->double "3.14") is the input value as a string instead of 
>> the result, I would need implement ::bigger-than-zero? and the rest of the 
>> checks as operating on strings, which would mean that I need in each check 
>> to transform the string into a double, which in my case is for performance 
>> reasons bad.
>>
>> On the other side I would be able to do the checks in a 2 step fashion, 
>> which goes against my understanding of the aim of clojure.check to be able 
>> to create through composition a single set of validations for a single value.
>> This also doesn't solve fully the problem with the multiple parsing of the 
>> first data type before the transformation: a parsing happens in the 
>> ::str->double step as well in the then explicit transformation step before 
>> the final checks based then on the actual double.
>>
>> So - is there a way to compose checks in a single one, where some of them 
>> change the underlying type of the value for subsequent validations (I may 
>> miss something in the Clojure documentation, and pointers to it would be 
>> very welcome)?
>>
>> If not - why s/conform returns the input value instead of the result value? 
>> As being on the caller site of s/conform I know what I passed and I know 
>> what s/conform would give me in the negative outcome of the validation.
>> If s/conform while iterating/recursing through the actual spec would apply 
>> the checks on returned values of previous checks instead always on the input 
>> - the problem would be solved, without negatives for the calling sites (as 
>> because of the previous sentence).
>> This could be solved for example (not sure if I see every detail yet, but at 
>> least as a pointer) in that in the clojure.spec code
>>
>> Spec
>> (conform* [_ x] (let [ret (pred x)]
>>   (if cpred?
>> ret
>> (if ret x ::invalid
>> the value of cpred? could be passed from the calling site (or if don't miss 
>> something be even per default true).
>>
>> In any case, I would be thankfull for any suggestions solving both proplems: 
>> how to express such checks in a good way and how to avoid reparsing values 
>> again and again (where the current example is for just strings to doubles, 
>> but the actual problem could involve any transformations on the value incl. 
>> type).
>>
>> With best regards
>> Plamen
>>
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to 

clojure.spec, conform and returned value

[plamen . usenet]

Hello,

I have a problem which is probably not in the spirit of clojure.spec as 
being a library for "only" checking/generating valid values, but of 
substantial practical value for my use case:

Let say I have a function for checking if a double precision number is 
parsable from a string (where possible performance penalties because of 
communicating non-parsable strings through try/catch in the implementation 
is not part of my question):


(defn str-parsable-double? [s]
  "Note the IMPLIED transformation from string to a double"
  (try
(Double/valueOf s)
(catch NumberFormatException e
  false)))


and have a spec using this function:

(s/def ::str->double str-parsable-double?)


We could actually chain checks for a single value as in:

(s/def ::str->double (s/and string? str-parsable-double?))

and we could check for the conformity of a value as per:

(s/conform ::str->double "3.14")
=> "3.14"
and 
(s/conform ::str->double "Pizza")
=> :clojure.spec/invalid

Until now everything is ok, now my the questions:

In my real world example I need a chain of a combination of checks and 
transformations in the following way:
::check1 ::check2 ::check-and-transform3 ::check-4-on-transformed-value 
::check-5-on-transformed value
or something like
(s/def ::transforming-check (s/and string? ::str->double ::bigger-than-zero? 
::smaller-than-pi ... etc ...))

As the result of 
(s/conform ::str->double "3.14") is the input value as a string instead of the 
result, I would need implement ::bigger-than-zero? and the rest of the checks 
as operating on strings, which would mean that I need in each check to 
transform the string into a double, which in my case is for performance reasons 
bad.

On the other side I would be able to do the checks in a 2 step fashion, which 
goes against my understanding of the aim of clojure.check to be able to create 
through composition a single set of validations for a single value.
This also doesn't solve fully the problem with the multiple parsing of the 
first data type before the transformation: a parsing happens in the 
::str->double step as well in the then explicit transformation step before the 
final checks based then on the actual double.

So - is there a way to compose checks in a single one, where some of them 
change the underlying type of the value for subsequent validations (I may miss 
something in the Clojure documentation, and pointers to it would be very 
welcome)?

If not - why s/conform returns the input value instead of the result value? As 
being on the caller site of s/conform I know what I passed and I know what 
s/conform would give me in the negative outcome of the validation.
If s/conform while iterating/recursing through the actual spec would apply the 
checks on returned values of previous checks instead always on the input - the 
problem would be solved, without negatives for the calling sites (as because of 
the previous sentence).
This could be solved for example (not sure if I see every detail yet, but at 
least as a pointer) in that in the clojure.spec code

Spec
(conform* [_ x] (let [ret (pred x)]
  (if cpred?
ret
(if ret x ::invalid
the value of cpred? could be passed from the calling site (or if don't miss 
something be even per default true).

In any case, I would be thankfull for any suggestions solving both proplems: 
how to express such checks in a good way and how to avoid reparsing values 
again and again (where the current example is for just strings to doubles, but 
the actual problem could involve any transformations on the value incl. type).

With best regards
Plamen





-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.