Re: Strange problem with input stream starting clojure 1.7.0-alpha6

[shlomivaknin]

@Kevin, yes, I did just that to test coll-reduce, thanks for helping me out!

@Alex, thanks for the detailed response, it sure demystified this error and 
gave a bunch of different solutions, awesome!


On Monday, March 7, 2016 at 5:17:45 PM UTC-8, Alex Miller wrote:
>
> Iterator, seqs, and chunking is indeed the key to this issue. The change 
> in question is http://dev.clojure.org/jira/browse/CLJ-1669, which made 
> iterator-seq's chunked. In general, this should not be problematic with 
> most Java iterators, however there is a iterator implementation pattern 
> where the identical object is returned (and mutated) on every call to 
> next() on the iterator. This is done in Java for performance reasons 
> (reuses a single mutable object rather than creating N objects). Because 
> iterator-seq now chunks, it will obtain 32 elements at a time, but they 
> will all be the identical object, so you'll see the state of the 32'nd 
> object for all of them.
>
> iterator-seq intentionally does not handle this iterator pattern (this is 
> in the docstring and was introduced via 
> http://dev.clojure.org/jira/browse/CLJ-1738). 
>
> If you need to control iteration to one at a time (non-chunked), then 
> there are several options:
>
> 1. Use loop/recur and explicitly handle each object one at a time. Sounds 
> like you've got that one.
>
> 2. Implement your own lazy-seq that does not chunk.
>
> (defn iter-seq [iter f]
>   (if (.hasNext iter)
> (lazy-seq
>   (cons (f (.next iter))
> (iter-seq iter f)
>
> 3. Use transducers which will also pull a single element at a time in 
> current transducible contexts. If you want to use it as a seq, this means 
> invoking 
>
> (defn transform [^ArchiveRecord r]
>   (let [header (.getHeader r)
>  mime (.getMimetype header)]
> (if (plain-text? mime)
>   (println "got " (.available r)
>
> (defn mapper-map [this ^Text key warc-value ^MapContext context]
>   (sequence (map transform) warc-value))
>
> 4. In this particular case, since you're only doing side effects, 
> something like dorun would maybe be better.
>
> 5. (WARNING - may be removed!) You can still access the old iterator-seq 
> impementation in the Java impl for now:
>
> (clojure.lang.IteratorSeq/create iter)
>
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Strange problem with input stream starting clojure 1.7.0-alpha6

[Alex Miller]

Iterator, seqs, and chunking is indeed the key to this issue. The change in 
question is http://dev.clojure.org/jira/browse/CLJ-1669, which made 
iterator-seq's chunked. In general, this should not be problematic with 
most Java iterators, however there is a iterator implementation pattern 
where the identical object is returned (and mutated) on every call to 
next() on the iterator. This is done in Java for performance reasons 
(reuses a single mutable object rather than creating N objects). Because 
iterator-seq now chunks, it will obtain 32 elements at a time, but they 
will all be the identical object, so you'll see the state of the 32'nd 
object for all of them.

iterator-seq intentionally does not handle this iterator pattern (this is 
in the docstring and was introduced via 
http://dev.clojure.org/jira/browse/CLJ-1738). 

If you need to control iteration to one at a time (non-chunked), then there 
are several options:

1. Use loop/recur and explicitly handle each object one at a time. Sounds 
like you've got that one.

2. Implement your own lazy-seq that does not chunk.

(defn iter-seq [iter f]
  (if (.hasNext iter)
(lazy-seq
  (cons (f (.next iter))
(iter-seq iter f)

3. Use transducers which will also pull a single element at a time in 
current transducible contexts. If you want to use it as a seq, this means 
invoking 

(defn transform [^ArchiveRecord r]
  (let [header (.getHeader r)
 mime (.getMimetype header)]
(if (plain-text? mime)
  (println "got " (.available r)

(defn mapper-map [this ^Text key warc-value ^MapContext context]
  (sequence (map transform) warc-value))

4. In this particular case, since you're only doing side effects, something 
like dorun would maybe be better.

5. (WARNING - may be removed!) You can still access the old iterator-seq 
impementation in the Java impl for now:

(clojure.lang.IteratorSeq/create iter)


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Strange problem with input stream starting clojure 1.7.0-alpha6

[Kevin Downey]

you can still structure you computation as a reduce, even if it is side
effectful.

(reduce (fn [sink record] (emit sink record) sink) sink source)

There is also a function introduced in 1.7 called "run!"
which is for processing a collection using reduce for side effects.

On 03/07/2016 04:49 PM, shlomivak...@gmail.com wrote:
> Thanks for your reply, 
> 
> ArchiveReader is actually an Iterator for ArchiveRecord
> (see 
> https://github.com/iipc/webarchive-commons/blob/master/src/main/java/org/archive/io/ArchiveReader.java#L51).
> I also tried to explicitly do (iterator-seq (.iterator warc-value)) but
> got the same "got 0" everywhere.
> 
> However, your remark about the lazy-seq was indeed very helpful. I tried
> to use a loop instead of doseq, and it did the trick!
> I also succeeded in using coll-reduce instead of the explicit loop.
> However, it seems odd to use it, since I am really only doing a
> side-effecting loop (emitting key/values for hadoop's reducer) rather
> than making a transformation..
> 
> Is there a nicer (idiomatic core) way to treat non-chunk-able java
> iterables other than an explicit loop for side-effects?
> 
> Thanks!
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to clojure+unsubscr...@googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.


-- 
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Strange problem with input stream starting clojure 1.7.0-alpha6

[shlomivaknin]

Thanks for your reply, 

ArchiveReader is actually an Iterator for ArchiveRecord 
(see 
https://github.com/iipc/webarchive-commons/blob/master/src/main/java/org/archive/io/ArchiveReader.java#L51).
 
I also tried to explicitly do (iterator-seq (.iterator warc-value)) but got 
the same "got 0" everywhere.

However, your remark about the lazy-seq was indeed very helpful. I tried to 
use a loop instead of doseq, and it did the trick!
I also succeeded in using coll-reduce instead of the explicit loop. 
However, it seems odd to use it, since I am really only doing a 
side-effecting loop (emitting key/values for hadoop's reducer) rather than 
making a transformation..

Is there a nicer (idiomatic core) way to treat non-chunk-able java 
iterables other than an explicit loop for side-effects?

Thanks!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Strange problem with input stream starting clojure 1.7.0-alpha6

[Kevin Downey]

Hard to say, I can't think of a change that would directly change how
the shaed code would work.

The "ArchiveReader" type hint on "warc-value" seems to be incorrect,
because it is used as a seq by "doseq". Assuming this is the correct
ArchiveReader
(http://crawler.archive.org/apidocs/org/archive/io/ArchiveReader.html) I
don't see how it could be a seq. So warc-value must be a seq, and is
likely to be lazy-seq. I wonder if you could be running in to a problem
with chunking or something(maybe you have a lazy-seq that needs to be
consumed strictly one after another, but a chunked seq would realize 32
at a time). "iterate" and "range" I think both had changes around then
so I would look at how that seq is constructed. If this is the issue,
you may want to look at using the clojure.core.protocols/CollReduce
protocol to process whatever the warc-value seq is based on instead of
using a seq.

On 03/07/2016 03:25 PM, shlomivak...@gmail.com wrote:
> Hey clojurians,
> 
> I am using a java library that reads WARC files (an internet archive
> format) to use with hadoop. 
> 
> I was recently motivated to upgrade this project's clojure from 1.6 to
> 1.8 (to be able to use the recent (wonderful!) cider), and I got quite a
> strange behavior, that I managed to reduce to a simple example (on
> github
> )
> 
>   
> (defn mapper-map [this ^Text key ^ArchiveReader warc-value ^MapContext
> context]
> 
> (doseq [^ArchiveRecord r warc-value]
>   (let [header (.getHeader r)
>   mime (.getMimetype header)]
>   (if (plain-text? mime)
>   (println "got " (.available r))
>   
> 
>   
> 
> 
> Using any clojure version prior to 1.7.0-alpha6 (meaning, alpha5 and
> below), this code works great, and I get plenty of different "got %d"
> printed to the console with different sizes.
> 
> However, upgrading to 1.7.0-alpha6 and above, I am getting constant "got
> 0" for every record in the file, and nothing (obviously) gets computed.
> 
> I tried to see if I can find the culprit
> using 
> https://github.com/clojure/clojure/compare/clojure-1.7.0-alpha5...clojure-1.7.0-alpha6
> and couldnt find an obvious problem. I thought I might ask the list for
> pointers before I deep dive into this any further. 
> 
> If you wish to help with this problem by checking it on your machine,
> you could clone https://github.com/vadali/warc-cc/tree/upgrade-clojure
> (use upgrade-clojure branch), get the example file into the root dir of
> the cloned project using 
> 
>s3cmd get
> s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-48/segments/1387345775423/wet/CC-MAIN-20131218054935-00092-ip-10-33-133-15.ec2.internal.warc.wet.gz
> 
> and run using lein test warc-cc.example.
> 
> Thanks!
> 
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to clojure+unsubscr...@googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.


-- 
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Strange problem with input stream starting clojure 1.7.0-alpha6

[shlomivaknin]

Hey clojurians,

I am using a java library that reads WARC files (an internet archive 
format) to use with hadoop. 

I was recently motivated to upgrade this project's clojure from 1.6 to 1.8 
(to be able to use the recent (wonderful!) cider), and I got quite a 
strange behavior, that I managed to reduce to a simple example (on github 

)


(defn mapper-map [this ^Text key ^ArchiveReader warc-value ^MapContext 
context] (doseq [^ArchiveRecord r warc-value]
(let [header (.getHeader r)
mime (.getMimetype header)]
(if (plain-text? mime)
(println "got " (.available r))




Using any clojure version prior to 1.7.0-alpha6 (meaning, alpha5 and 
below), this code works great, and I get plenty of different "got %d" 
printed to the console with different sizes.

However, upgrading to 1.7.0-alpha6 and above, I am getting constant "got 0" 
for every record in the file, and nothing (obviously) gets computed.

I tried to see if I can find the culprit 
using 
https://github.com/clojure/clojure/compare/clojure-1.7.0-alpha5...clojure-1.7.0-alpha6
 
and couldnt find an obvious problem. I thought I might ask the list for 
pointers before I deep dive into this any further. 

If you wish to help with this problem by checking it on your machine, you 
could clone https://github.com/vadali/warc-cc/tree/upgrade-clojure (use 
upgrade-clojure branch), get the example file into the root dir of the 
cloned project using 

   s3cmd get 
s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-48/segments/1387345775423/wet/CC-MAIN-20131218054935-00092-ip-10-33-133-15.ec2.internal.warc.wet.gz

and run using lein test warc-cc.example.

Thanks!


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Ubergraph 0.2.0

[Mark Engelberg]

Just issued a quick bugfix release, version 0.2.1, for an issue with
`find-edges` reported and patch provided by github/lomin.

Thanks lomin!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

clojure .hashCode tests failure with ibm java.

[Alex Miller]

This is a known issue and we agree that the error is in the tests. However, we 
have not prioritized fixing it as it affects so few users (very few people run 
the Clojure build itself on the IBM jdk).

You can see more on the ticket http://dev.clojure.org/jira/browse/CLJ-1678

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Doing matrix arithmetic with complex numbers

[Al Matthews]

+1

On Sat, Mar 5, 2016, 22:12 Mars0i  wrote:

> You might want to post this on the Numerical clojure group as well or
> instead.  I've never tried using complex numbers with core.matrix.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-- 

Alfred S Matthews

+1 337 214 4688

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

clojure .hashCode tests failure with ibm java.

[raminder sodhi]

Hi,
I was trying to build clojure with ibm versions of java and noticed that 
the following tests fail.

To debug i wrote the following program:
import java.math.BigInteger;

public class testHash {
   public static void main(String[] args) {
  BigInteger i = new BigInteger("1");
  BigInteger j = new BigInteger("9223372039002259457");
  //String j = new String("9223372039002259457N");
  //String j = new String("9223372039002259457N");
  System.out.println(i.hashCode());
  System.out.println(j.hashCode());
   }
}

which has the output:
1
33


The hashCode() method in Java only promises to report the same hash value 
for the same object, and to report the same value for two objects which 
equals() says are equal. It does not make any promise about the value 
itself, or about consistency with alternative JVM implementations.
Reference: 
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Object.html#hashCode%28%29

So I kind of see the following as a wrong test.  Let me know if i'm wrong.



The failing tests:
https://github.com/clojure/clojure/blob/master/test/clojure/test_clojure/control.clj

(testing "test correct behavior on hash collision"
;; case uses Java .hashCode to put values into hash buckets.
(is (== (.hashCode 1) (.hashCode 9223372039002259457N)))
(are [result input] (= result (case input
1 :long
9223372039002259457N :big
:else))
:long 1
:big 9223372039002259457N
:else 4294967296
:else 2)
(are [result input] (= result (case input
9223372039002259457N :big
1 :long
:else))
:long 1
:big 9223372039002259457N
:else 4294967296
:else 2)
(are [result input] (= result (case input
0 :zero
-1 :neg1
2 :two
:oops :OOPS))
:zero 0
:neg1 -1
:two 2
:OOPS :oops)
(are [result input] (= result (case input
1204766517646190306 :a
1 :b
-2 :c
:d))
:a 1204766517646190306
:b 1
:c -2
:d 4294967296
:d 3))
(testing "test warn for hash collision"
(should-print-err-message
#"Performance warning, .*:\d+ - hash collision of some case test constants; 
if selected, those entries will be tested sequentially..*\r?\n"
(case 1 1 :long 9223372039002259457N :big 2)))
(testing "test constants are *not* evaluated"
(let [test-fn
;; never write code like this...
#(case %
(throw (RuntimeException. "boom")) :piece-of-throw-expr
:no-match)]
(are [result input] (= result (test-fn input))
:piece-of-throw-expr 'throw
:piece-of-throw-expr '[RuntimeException. "boom"]
:no-match nil

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Prismatic Schema - Why is "both" deprecated?

[Jason Wolfe]

Yes, that use case (two different non-predicate schemas) is no longer
supported.  While superficially simple, it added a fair bit of
complexity and had corner cases that are not present with
`conditional` or `constrained`.  Moreover, in our experience the
primary use case for `both` by far was with a predicate, and in the
other cases there is typically a reasonable alternative.   Can you
elaborate on what your use case looks like?

Thanks, Jason

On Mon, Mar 7, 2016 at 11:41 AM, JvJ  wrote:
> In this example, one of the schemas is a predicate schema.  What if neither
> of them are?
>
>
> On Sunday, 6 March 2016 18:48:21 UTC-8, Jason Wolfe wrote:
>>
>> In place of (s/both long (s/pred odd?)) you can do (s/conditional odd?
>> long), but (s/constrained long odd?) probably provides better error messages
>> (since it validates long before odd?). I think there are some examples in
>> the readme.
>>
>> If this isn't what you're looking for, can you please provide some more
>> details of your use case?
>>
>> Thanks,
>> Jason
>>
>>
>>
>> On Sunday, March 6, 2016 at 5:38:35 AM UTC+5:30, JvJ wrote:
>>>
>>>
>>> I've noticed that there is the function "both" in Schema.
>>>
>>> Both says that it can be replaced by conditional, but I'm not sure
>>> exactly how to go about doing this.
>>>
>>> Can someone provide an example?
>>>
>>> Thanks
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/1Xfr_VS2yV4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.