Re: Clojure Packaging Team BoF @DebConf - Aug 26 (Thu): 17:00 UTC (Online)

[Chris Nuernberger]

That is very cool!!

On Wed, Aug 25, 2021, 6:01 PM Leandro Doctors  wrote:

> FYI
> --Leandro
>
> ***
>
> Clojure Packaging Team BoF
> Speaker: Elana Hashman
>
> Let’s gather the Debian Clojure community so we can find more
> maintainers, discuss Clojure policy, prioritize applications to be
> introduced into Debian, and talk about open ITPs for Clojure packages.
> This year, we have a GSoC student working on packaging some essential
> tools! ehashman will provide an introduction to Clojure packaging
> structure and policy if requested.
>
> Event details:
> https://debconf21.debconf.org/talks/12-clojure-packaging-team-bof/
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/CAAn03x50pE3p_4sT75xuwPHoVdu6XmXO6pfbU5bUQWkBbAUfQw%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJt7%2B-sULEOves6833PjHcer1B5Md6kC1gW0bEMAeN8e3g%40mail.gmail.com.

Re: How to get a value of a var in Java

[Chris Nuernberger]

Vars extend IDeref so one way to get the value of a var is to call .deref
on it:

user> (def a)
#'user/a
user> (defn ab [] 1)
#'user/ab
user> (.deref (clojure.lang.RT/var "user" "a"))
2
user> (.deref (clojure.lang.RT/var "user" "ab"))
#function[user/ab]



On Fri, Jun 18, 2021 at 7:12 AM ru  wrote:

> Thanx Alex!
>
> Alas, with defn it works, with def - not! :(
>
> пятница, 18 июня 2021 г. в 15:31:03 UTC+3, Alex Ott:
>
>> does this answer your q:
>> https://stackoverflow.com/questions/2181774/calling-clojure-from-java/2182608#2182608
>> ?
>>
>> On Fri, Jun 18, 2021 at 12:53 PM ru  wrote:
>>
>>> Dear Clojure users and team!
>>> Citation 1 from Clojure documentation:
>>>
>>> "Calling Clojure From Java..
>>> IFn plus = Clojure.var("clojure.core", "+");
>>> plus.invoke(1, 2);.."
>>>
>>> With functions all well but with def..
>>>
>>> Citation 2:
>>>
>>> "(def *symbol* *doc-string*? *init*?)
>>>
>>> Creates and interns or locates a global var
>>>  with the name of *symbol* and a
>>> namespace of the value of the current namespace (*ns*). "
>>>
>>> I can not figure out how to get the value of the symbol, created with
>>> def construct in Clojure and, for example, print it with System.out.println
>>> in Java.
>>>
>>> I tried everything, nothing works! :(
>>>
>>> Any help would be greatly appreciated.
>>>
>>> Sincerely,
>>>
>>>   Ru
>>>
>>>
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/clojure/fcbd3883-705a-4180-9b99-7ccad64a09afn%40googlegroups.com
>>> 
>>> .
>>>
>>
>>
>> --
>> With best wishes,Alex Ott
>> http://alexott.net/
>> Twitter: alexott_en (English), alexott (Russian)
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/ff4cb2f0-b121-4c54-b060-a1e64fbb0e4an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJumuL42Zqh29LnsDo6BUUQTjT8gbXrsW6%3DAOPRoOBkNNA%40mail.gmail.com.

Re: Routing for a non-web-app

[Chris Nuernberger]

For 'routing' specifically, if you want a more formal system we used state
machines in the past and recently Clojure has a really nice additional
possibility there - https://github.com/lucywang000/clj-statecharts.

For general UI programming if you are using javafx then I really like cljfx
- https://github.com/cljfx/cljfx.

On Mon, Apr 19, 2021 at 1:46 AM Orestis Markou  wrote:

> As far as I know, reitit could probably do what you want. It works on
> ClojureScript too.
>
> On 19 Apr 2021, at 01.58, Blake Watson  wrote:
>
> Howdy,
>
> Writing a desktop app and thinking describing routes is the way to go, as
> far as setting up the flow through the program. There are quite a few
> Clojure routing libraries but most seem to assume it's a web backend
> (naturally), and the ones that don't seem to be ClojureScript (which I
> can't use here).
>
> I could just use a multi-method and roll my own, of course, but I thought
> it might be useful to try out an existing library.
>
> ===Blake===
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/CAJAnwPnBTuHsbDA_OgGSAt1j_B_tL%2B%3DnCVsigwZrAzD_g9%3D%2BsA%40mail.gmail.com
> 
> .
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/DEF9D4FD-EE14-4855-BB97-A9297DC9DD97%40orestis.gr
> 
> .
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJvUMHzfUN3a2KaJv0YZcYubJDbSvnJ2LLd9D0Fy-T3djA%40mail.gmail.com.

[ANN] - libpython-clj2 beta 1

[Chris Nuernberger]

Good morning Clojurians,

We have an important new release of libpython-clj which brings support for
JDK-16's [ffi interface](https://openjdk.java.net/jeps/191) and support for
32 bit systems.  We built this on a brand new [ffi layer](
https://cnuernber.github.io/dtype-next/tech.v3.datatype.ffi.html) in
dtype-next which virtualizes the actual FFI system and then uses
codegeneration to create the exact types and binding sites required for a
given subsystem - JNA or JDK-16 at this pointer - all mostly transparently
to the user.

Interestingly, JDK-16's FFI pathway **is** much faster than JNA's
directmapped pathway but it is also more brittle; when I attempted to load
numpy things failed but aside from that tests passed and I imagine if I
wanted to mess around with LD_LIBRARY_PATH I could get everything to work.

The virtualization of the FFI pathway also opens the door to other python
binary library providers such as directly linking a graal native executable
to libpython shared library or using javacpp's version of python.

I am really proud to bring you this as I see it as an important step to
both simplify libpython-clj and to provide serious future proofing so we
can rely on the library for some time moving forward.  Enjoy!

Chris

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJs7pff7u_%3DNwxAPR85svUh5d4uT7%2B0JFvL%2BeeJQ37VvnA%40mail.gmail.com.

[ANN] Clojure High Performance Computing Library

[Chris Nuernberger]

Good Day Clojurians!

I am very proud to announce cnuernber/dtype-next {:mvn/version "6.00"}
. This major rewrite of our
Clojure high performance computing library targeting higher performance,
wider platform support, and reduced code footprint.dtype-next
 is a Clojure library that adds a
coherent C/java integration, lazy functional, and high performance
numerics, N-dimensional array programming, index space algorithms and high
performance reductions. It works on JDK-8 through 14 and has partial JDK-16
and Graal Native support.

Enjoy :-)

Chris

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJueJoxxbKAn_Rafsn%2ByaZWf%2Bzj-f7gvFJPruhN1hWQGAg%40mail.gmail.com.

Re: Calling Native C-code from Clojure

[Chris Nuernberger]

You are most welcome :-).  For numerical computation, libjulia-clj
 may interest you :-).

On Sat, Dec 5, 2020 at 8:06 AM John Doe  wrote:

> Thank you Chris. It's very helpful.
>
> The approach you took for dtype-next is very interesting. This library is
> closer to my work (numerical computation). Also, the efficient use of Java
> heap is very interesting although I admit I don't know enough about it.
> This reminds me of Rich Hickey's emphasis of understanding data structures.
>
> On Friday, December 4, 2020 at 4:52:25 PM UTC+1 ch...@techascent.com
> wrote:
>
>> Ngwua,
>>
>> Looking through the caesium bindings just mentioned, it appears the
>> author dynamically creates an interface and then you can simply ask the
>> native library to instantiate a concrete implementation of that interface
>> .
>> So you do not always have to create a class with 'native' methods.
>>
>> Integrating with C libraries is part of the reason I wrote dtype-next
>>  which has the ability to
>> efficiently write/read from either java heap or native heap storage.
>> dtype-next has no dependency on either JNA or JNR.
>>
>> Hope this helps,
>>
>> Chris
>>
>> On Fri, Dec 4, 2020 at 7:53 AM John Doe  wrote:
>>
>>> Hello Everyone,
>>>
>>> *Motivation:* I want to invoke C-code functions from Clojure: writing
>>> Java functions in Clojure.
>>>
>>> *Plan of Attack:* Rather than go with the JNI approach I want to
>>> interface with C-Library via JNR-FFI (https://github.com/jnr/jnr-ffi)
>>>
>>> Based on some example of using JNR-FFI
>>>
>>> https://github.com/jnr/jnr-ffi-examples/blob/master/getpid/src/main/java/getpid/Getpid.java
>>>
>>> https://www.programcreek.com/java-api-examples/?api=jnr.ffi.LibraryLoader
>>>
>>> The wrapper in Java might look like
>>> ```
>>> package blahblah;
>>>
>>> public class BlahNative {
>>> private static int[] intDummy;
>>> private static double[] doubleDummy;
>>>
>>> public BlahNative() {
>>> }
>>>
>>> public static native void native_a(char var0, char var1, int var2, int
>>> var3, int var4, double[] var7, int var8, int var9);
>>>
>>> static {
>>> lib = LibraryLoader
>>> .create(LibSodium.class)
>>> .search("/usr/local/lib")
>>> .search("/opt/local/lib")
>>> .search("/usr/lib")
>>> .search("/lib")
>>> .load(LIBRARY_NAME);
>>> initializeLibrary(lib);
>>> }
>>> }
>>> ```
>>>
>>> Question: For my package
>>> ```
>>> (defproject mypkg
>>> :dependencies [[org.clojure/clojure "1.10.1"]
>>>[com.github.jnr/jnr-ffi "2.2.0"]])
>>> ```
>>> How do I write the wrapper for the native functions in
>>> ```
>>> (ns mypkg.native
>>> (:import [jnr.ffi LibraryLoader])
>>> (:import [jnr.ffi.annotations IgnoreError])
>>> (:import [jnr.ffi.provider FFIProvider]))
>>> ```
>>> For the Java interop I am assuming the above namespace should be based
>>> on gen-class (http://clojure-doc.org/articles/language/interop.html).
>>>
>>> What is not clear to me is:
>>> *1. How does one create the `native` method* (since most (defn foo) are
>>> static methods)?
>>> *2. How do I access the native functions after loading* the Native
>>> Library (LibraryLoader.load(LIBRARY_NAME))?
>>>
>>> I am neither a C nor a Java expert so any guidance will be much
>>> appreciated.
>>>
>>> Thanks,
>>> Ngwua
>>>
>>> --
>>>
>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/clojure/3bd05d7a-297e-49cf-86d4-aec4a88c83cbn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group 

Re: Calling Native C-code from Clojure

[Chris Nuernberger]

Ngwua,

Looking through the caesium bindings just mentioned, it appears the author
dynamically creates an interface and then you can simply ask the native
library to instantiate a concrete implementation of that interface
.
So you do not always have to create a class with 'native' methods.

Integrating with C libraries is part of the reason I wrote dtype-next
 which has the ability to
efficiently write/read from either java heap or native heap storage.
dtype-next has no dependency on either JNA or JNR.

Hope this helps,

Chris

On Fri, Dec 4, 2020 at 7:53 AM John Doe  wrote:

> Hello Everyone,
>
> *Motivation:* I want to invoke C-code functions from Clojure: writing
> Java functions in Clojure.
>
> *Plan of Attack:* Rather than go with the JNI approach I want to
> interface with C-Library via JNR-FFI (https://github.com/jnr/jnr-ffi)
>
> Based on some example of using JNR-FFI
>
> https://github.com/jnr/jnr-ffi-examples/blob/master/getpid/src/main/java/getpid/Getpid.java
>
> https://www.programcreek.com/java-api-examples/?api=jnr.ffi.LibraryLoader
>
> The wrapper in Java might look like
> ```
> package blahblah;
>
> public class BlahNative {
> private static int[] intDummy;
> private static double[] doubleDummy;
>
> public BlahNative() {
> }
>
> public static native void native_a(char var0, char var1, int var2, int
> var3, int var4, double[] var7, int var8, int var9);
>
> static {
> lib = LibraryLoader
> .create(LibSodium.class)
> .search("/usr/local/lib")
> .search("/opt/local/lib")
> .search("/usr/lib")
> .search("/lib")
> .load(LIBRARY_NAME);
> initializeLibrary(lib);
> }
> }
> ```
>
> Question: For my package
> ```
> (defproject mypkg
> :dependencies [[org.clojure/clojure "1.10.1"]
>[com.github.jnr/jnr-ffi "2.2.0"]])
> ```
> How do I write the wrapper for the native functions in
> ```
> (ns mypkg.native
> (:import [jnr.ffi LibraryLoader])
> (:import [jnr.ffi.annotations IgnoreError])
> (:import [jnr.ffi.provider FFIProvider]))
> ```
> For the Java interop I am assuming the above namespace should be based on
> gen-class (http://clojure-doc.org/articles/language/interop.html).
>
> What is not clear to me is:
> *1. How does one create the `native` method* (since most (defn foo) are
> static methods)?
> *2. How do I access the native functions after loading* the Native
> Library (LibraryLoader.load(LIBRARY_NAME))?
>
> I am neither a C nor a Java expert so any guidance will be much
> appreciated.
>
> Thanks,
> Ngwua
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/3bd05d7a-297e-49cf-86d4-aec4a88c83cbn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJuH5i3Py17byfRR5zoiXUXFWzk-xLtX6iLzGRmnoHyb6g%40mail.gmail.com.

Clojure integration with JuliaLang

[Chris Nuernberger]

Good afternoon/morning Clojurians,


Over the thanksgiving weekend I finished up enough of a library for calling
Julia from Clojure to start talking about it publicly.

Julia sits kind of at the pinnacle of the numerics stack. It is a
generalized very high performance language specifically designed to make
mathematical and scientific computing clearer, more enjoyable, and more
correct. In the techascent stack, it sits just *after* something like
Smile's BLAS binding but just *before* TVM. Unlike a specific blas binding,
you can write bespoke code in Julia and it is highly optimized for you.
Unlike TVM, the language is Turing complete.

Julia has the best solvers known to computer science and it has a fully
differentiable computing stack. I think it points the way forward in terms
of dense and sparse style numeric computing. It has great metaprogramming
capabilities and a community of extremely capable people pushing it forward.

Pairing Julia tightly with Clojure allows us to use their vast libraries of
scientific computing and gives us an avenue to take advantage of truly
cutting edge numeric research directly. I would like Julia people to enjoy
using Clojure and vice versa.

This is absolutely a hair-on-fire early release. I put this forward in
order to just let everyone know it is concretely possible and to open the
door for people who may be interested in helping out.

https://github.com/cnuernber/libjulia-clj

Enjoy,

Chris

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJsBpPsMFUXN9UAe2MBFtv0wR80%3DnEsxB%2BAQQasj_Tueiw%40mail.gmail.com.

Re: Clojure as first language

[Chris Nuernberger]

Oh, well then I don't see what you are waiting for :-).

Here are some interesting and more stats-focused libraries that may be
interesting to you -

* kixi stats  - Clojury statistics
- written by Henry Gardner, the author of the aforementioned Clojure For
Data Science

* fastmath  - Carefully
chosen and curated fast JVM mathematical primitives
* clojisr  - R <-> Clojure bridge
* Anglican  -  a
probabilistic programming language
* Bayadera  - MKL, GPU enhanced
probabilistic programming system.

Enjoy :-)

On Mon, Sep 28, 2020 at 11:57 AM Baye  wrote:

> Hi Chris, Thanks! I will check out the mentioned resources.
> Just to be clear, the only language I know well is Stata. I am still a
> very new to python, so I don't have any baggage to take with me as I have
> not invested enough time.
> Given I am philosophically convinced of the long term benefits of Clojure
> from talks, my only apprehension was whether I will be able to do most
> things in Clojure as in Python. But my interest, in programming is not
> limited to data science/ML...I  am potentially interested in building apps
> (web/desktop, etc) for potential future projects in education, health etc..
> On Monday, September 28, 2020 at 8:39:06 PM UTC+3 ch...@techascent.com
> wrote:
>
>> There are hybrid options available in the form of
>> https://github.com/clj-python/libpython-clj -- I am one of the primary
>> authors of this tool.
>>
>> One pathway perhaps is to use clojure to do your scraping and
>> orchestration (and frontend display) and just use python from command line
>> scripts to do some ml.
>>
>> For Clojure and data science there is SciCloj: https://scicloj.github.io/
>>
>> My opinion is that learning Clojure independent of data science  is a
>> worthy and substantial task - functional programming, the JVM, Java,
>> Clojurescript, Reagent are all pretty big subjects.  libpython-clj has a
>> new-to-clojure
>> 
>> page that lists some resources for helping with this.
>>
>> Happy to help more and really happy to see new people.  I think Gary's
>> response is spot on and just wanted to elaborate that we have tools that
>> are specifically designed for helping people transition from Python to
>> Clojure without needing to walk away from their Python knowledge base.
>>
>> Chris
>>
>> On Mon, Sep 28, 2020 at 11:08 AM Baye  wrote:
>>
>>> Got it. Thanks again for your time!
>>>
>>> On Monday, September 28, 2020 at 8:05:32 PM UTC+3 Gary Trakhman wrote:
>>>
 Not only the language and the techniques you intend to apply, but the
 larger language and library ecosystem.  For most people, Clojure is best
 used when you
 have familiarity with the JVM and existing java core libraries and
 3rd-party libraries, and some IDE/editor environment with enough features
 to be productive, but
 those are extra things to learn.  Something like Scheme/Racket used for
 SICP is a great teaching language because all those decisions are made for
 you and
 you can focus on the abstract content in the book instead of getting
 bogged down in tooling.

 As for data-science/ML, it has good tools, it's just not the most
 widely used language by data scientists.
 Check out Clojure for data science:
 https://www.amazon.com/Clojure-Data-Science-Henry-Garner/dp/1784397180

 For the most part, I'd expect production-scale data science tools to
 wrap java ones with a better/clojurey interface.
 There's some other tools I'm not familiar with, eg
 https://github.com/originrose/cortex, but there are definitely people
 doing data science in clojure.

 On Mon, Sep 28, 2020 at 12:55 PM Soule S  wrote:

> Thank you for your insight. By " learning more than one thing at a
> time", do you mean data science/ML and the language itself? If so, then I
> want to clarify, I have a solid foundation in data science; I am trained 
> in
> the most advanced topics in econometrics (Statistics). My apprehension is
> mainly whether closure has good tools to do those. I know Python does.
>
> Thanks again
>
> On Mon, Sep 28, 2020 at 7:40 PM Gary Trakhman 
> wrote:
>
>> Clojure is a fine language to learn on, but there are going to be
>> some complex details along the way. I think there are 2 approaches
>> to entry into programming that you might consider.
>>
>> You can start with the basic fundamentals of computation and work up
>> into software engineering. For that, clojure is
>> very similar to a teaching language, 'scheme', and people can work
>> through books like 

Re: Clojure as first language

[Chris Nuernberger]

There are hybrid options available in the form of
https://github.com/clj-python/libpython-clj -- I am one of the primary
authors of this tool.

One pathway perhaps is to use clojure to do your scraping and orchestration
(and frontend display) and just use python from command line scripts to do
some ml.

For Clojure and data science there is SciCloj: https://scicloj.github.io/

My opinion is that learning Clojure independent of data science  is a
worthy and substantial task - functional programming, the JVM, Java,
Clojurescript, Reagent are all pretty big subjects.  libpython-clj has a
new-to-clojure

page that lists some resources for helping with this.

Happy to help more and really happy to see new people.  I think Gary's
response is spot on and just wanted to elaborate that we have tools that
are specifically designed for helping people transition from Python to
Clojure without needing to walk away from their Python knowledge base.

Chris

On Mon, Sep 28, 2020 at 11:08 AM Baye  wrote:

> Got it. Thanks again for your time!
>
> On Monday, September 28, 2020 at 8:05:32 PM UTC+3 Gary Trakhman wrote:
>
>> Not only the language and the techniques you intend to apply, but the
>> larger language and library ecosystem.  For most people, Clojure is best
>> used when you
>> have familiarity with the JVM and existing java core libraries and
>> 3rd-party libraries, and some IDE/editor environment with enough features
>> to be productive, but
>> those are extra things to learn.  Something like Scheme/Racket used for
>> SICP is a great teaching language because all those decisions are made for
>> you and
>> you can focus on the abstract content in the book instead of getting
>> bogged down in tooling.
>>
>> As for data-science/ML, it has good tools, it's just not the most widely
>> used language by data scientists.
>> Check out Clojure for data science:
>> https://www.amazon.com/Clojure-Data-Science-Henry-Garner/dp/1784397180
>>
>> For the most part, I'd expect production-scale data science tools to wrap
>> java ones with a better/clojurey interface.
>> There's some other tools I'm not familiar with, eg
>> https://github.com/originrose/cortex, but there are definitely people
>> doing data science in clojure.
>>
>> On Mon, Sep 28, 2020 at 12:55 PM Soule S  wrote:
>>
>>> Thank you for your insight. By " learning more than one thing at a
>>> time", do you mean data science/ML and the language itself? If so, then I
>>> want to clarify, I have a solid foundation in data science; I am trained in
>>> the most advanced topics in econometrics (Statistics). My apprehension is
>>> mainly whether closure has good tools to do those. I know Python does.
>>>
>>> Thanks again
>>>
>>> On Mon, Sep 28, 2020 at 7:40 PM Gary Trakhman 
>>> wrote:
>>>
 Clojure is a fine language to learn on, but there are going to be some
 complex details along the way. I think there are 2 approaches
 to entry into programming that you might consider.

 You can start with the basic fundamentals of computation and work up
 into software engineering. For that, clojure is
 very similar to a teaching language, 'scheme', and people can work
 through books like SICP:
 https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book.html
 with it.
 If you follow that strategy, you'll eventually have a good foundation
 of understanding to build on, but it might not feel very useful in the
 first year.

 Another strategy is to hack up small demos of individual techniques,
 like numerical methodologies or data visualizations.  This would go more
 along the data-science/ML track. While it can be done in clojure and
 there are books on the topic, you might have an easier time in python
 if you're going down that path.

 I would say if you start with clojure, you're going to be learning more
 than one thing at a time and the learning curve is steeper. Whether
 that's worth the cost depends on your goals and market conditions.


 On Mon, Sep 28, 2020 at 12:13 PM Baye  wrote:

> I am new to programming. I have started to learn C++ and even some
> python
> My background is Economics. I have strong quantitative background
> (Math and Econ) and I know STATA.
>
> In general, I would like to use programming
> (1) for ML/AI in economics topics+other utilities such as webcraping,
> and other data driven analyses
> (2) for it, fintech matchmaking platforms
>
> In particular, I just recently discovered I love programming. I am old
> (35) as a beginner but I have set a goal to become a fully capable
> programmer by 40...Hopefully I can learn 2 languages well enough by then.
>
> If jobs opportunities are present then, I can switch careers although
> I still like my career path. I could always use my programming skills for
> 

[ANN] - tech.ml.dataset upgrades + memory map arrow blogpost

[Chris Nuernberger]

We have been busy here working with our little dataset library and I wanted
to talk about some of the upgrades that I think are important/interesting.

We have first class support of Apache Arrow now which means I took the time
to actually understand, byte-by-byte, the binary on-disk format. I also
found a memory mapping library that I think is great, larray
.

Arrow files are really sequences of datasets and we have a brand new
namespace

which will grow over time devoted to really large (multiple GB, out of
memory) reductions over sequences of datasets which have competitive
performance characteristics with anything out there.

Lastly, we have a (great!) blog post exploring memory mapping, apache
arrow, and the tech.datatype bindings

to larray. It specifically highlights how nice Clojure is for exploring
binary file formats — doing this brought back memories of doing similar
things in C++ and wow, with Clojure I get the same performance and I can
actually see what I am working with! Much appreciation to Alex, Rich, and
the team!

https://techascent.com/blog/memory-mapping-arrow.html

Enjoy!

Chris

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJsz1T%3D2Gu%2BVpxdy02QK8Ap2n1PxkmRy%2BCAJqWKepKwmVQ%40mail.gmail.com.

Re: [ANN] tech.ml.dataset - 2.0

[Chris Nuernberger]

Thank you Alexandre!  I have to admit it is a *ton* of work.  I think there
are lots of good pathways literally every direction such as simplifying the
numerics layer (tech.datatype), potentially getting a subset working on
graalvm-native, zero-copy conversion when possible for parquet and arrow
(totally possible in lots of cases), etc. etc; it just depends on what
seems like it provides the most value to everyone.

Plus learning just exactly how to use this system is a thing; it is complex
as are numpy, pandas, data.table   Bridging between Clojure and APL,
and C puts this in a unique position.

That being said, Thomaz has released tablecloth
<https://github.com/scicloj/tablecloth> which has a more advanced dataset
api based on the primitives in tech.ml.dataset with some great documentation
<https://scicloj.github.io/tablecloth/index.html>.

On Mon, Jun 15, 2020 at 1:48 PM Alexandre Almosni <
alexandre.almo...@gmail.com> wrote:

> Congratulations. This is really a great effort and something we really
> needed. I hope the community takes this as the base layer for data science
> and we can build on your efforts, expand the documentation, etc.
>
>
>
> On Monday, June 15, 2020 at 5:50:52 PM UTC+1, Chris Nuernberger wrote:
>>
>> Good morning Clojurians :-)
>>
>> It is with much pride that I announce version 2.0 of tech.ml.dataset
>> <https://github.com/techascent/tech.ml.dataset>, our library that maps
>> powerful concepts from libraries like Pandas and data.table into Clojure
>> using functional paradigms. This data frame
>> <https://github.com/mobileink/data.frame/wiki/What-is-a-Data-Frame%3F>
>> library has unified loading from csv, tsv, xlsx, xls, Apache parquet,
>> Apache arrow (.feather), sql, json and sequences of maps as well as
>> efficient cpu and memory
>> <https://gist.github.com/cnuernber/26b88ed259dd1d0dc6ac2aa138eecf37>
>> performance. Finally, because the dataset knows the datatype of each
>> column, you can interoperate with schema-ful things like SQL
>> <https://github.com/techascent/tech.ml.dataset.sql> without writing down
>> the schema.
>>
>>
>> user> (require '[tech.ml.dataset :as ds])
>> nil
>> user> (-> (ds/->dataset "https://vega.github.io/vega/data/stocks.csv;)
>>   
>> (ds/descriptive-stats))https://vega.github.io/vega/data/stocks.csv: 
>> descriptive-stats [3 10]:
>>
>> | :col-name |  :datatype | :n-valid | :n-missing |   :min |  
>> :mean | :mode |   :max | :standard-deviation | :skew |
>> |---||--||||---||-|---|
>> |  date | :packed-local-date |  560 |  0 | 2000-01-01 | 
>> 2005-05-12 |   | 2010-03-01 | |   |
>> | price |   :float32 |  560 |  0 |  5.970 |  
>> 100.7 |   |  707.0 |   132.6 | 2.413 |
>> |symbol |:string |  560 |  0 ||  
>>   |  MSFT || |   |
>>
>> Data science is (still) alive and well in Clojure and the JVM. Stepping
>> back and considering python bindings
>> <https://github.com/clj-python/libpython-clj>, R bindings
>> <https://github.com/scicloj/clojisr>, smile <https://haifengl.github.io/>,
>> the next-gen blas/numerics library Neanderthal
>> <https://github.com/uncomplicate/neanderthal> and the exceptionally
>> powerful saite science platform <https://github.com/jsa-aerial/saite>,
>> we have really come a long way in the last year!
>>
>> Thanks and enjoy :-)
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/d2063089-7985-4de7-8c40-fd178667dcbbo%40googlegroups.com
> <https://groups.google.com/d/msgid/clojure/d2063089-7985-4de7-8c40-fd178667dcbbo%40googlegroups.com?utm_medium=email_source=footer&

[ANN] tech.ml.dataset - 2.0

[Chris Nuernberger]

Good morning Clojurians :-)

It is with much pride that I announce version 2.0 of tech.ml.dataset
, our library that maps
powerful concepts from libraries like Pandas and data.table into Clojure
using functional paradigms. This data frame

library has unified loading from csv, tsv, xlsx, xls, Apache parquet,
Apache arrow (.feather), sql, json and sequences of maps as well as
efficient cpu and memory

performance. Finally, because the dataset knows the datatype of each
column, you can interoperate with schema-ful things like SQL
 without writing down
the schema.


user> (require '[tech.ml.dataset :as ds])
nil
user> (-> (ds/->dataset "https://vega.github.io/vega/data/stocks.csv;)
  (ds/descriptive-stats))https://vega.github.io/vega/data/stocks.csv:
descriptive-stats [3 10]:

| :col-name |  :datatype | :n-valid | :n-missing |   :min
|  :mean | :mode |   :max | :standard-deviation | :skew |
|---||--||||---||-|---|
|  date | :packed-local-date |  560 |  0 | 2000-01-01
| 2005-05-12 |   | 2010-03-01 | |   |
| price |   :float32 |  560 |  0 |  5.970
|  100.7 |   |  707.0 |   132.6 | 2.413 |
|symbol |:string |  560 |  0 |
||  MSFT || |   |

Data science is (still) alive and well in Clojure and the JVM. Stepping
back and considering python bindings
, R bindings
, smile ,
the next-gen blas/numerics library Neanderthal
 and the exceptionally
powerful saite science platform , we
have really come a long way in the last year!

Thanks and enjoy :-)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJuDaq1719%3DXfXvPUP3gU4SdGLRNSECqXE4zOk3Vi%3D9qfA%40mail.gmail.com.

[ANN] - libpython-clj news

[Chris Nuernberger]

Good morning Clojurians from snowy Boulder, CO -


libpython-clj has been undergoing quite a bit of development.  There is now
extensible datafy/nav support for python modules and classes.  We have much
better (but still imperfect) environment support.  We have a clj-template
so you can use Sean Corfield's clj-new system.

We also spent some time profiling things to make sure we pay as little of a
cost as possible for crossing the language boundaries.  The cost is still
there of course but we reduced it by a factor of at least 3 or so.

We moved GC to be cooperatively done in the python thread.  So if you want
to use python from only on clojure thread that is now possible.

We have a github organization so we can expand the ecosystem of Clojure and
Python-related goodness outside the scope of libpython-clj.

Carin Meier has been doing quite a few demonstration projects using
Nextjournal.  I really suggest checking those out as you can simply hit
'remix' and start playing with the code.  If you haven't checked out
Nextjournal in general then by all means, give it a try.  It is a great
system for online live-notebook coding.

* Changelog -
https://github.com/clj-python/libpython-clj/blob/master/CHANGELOG.md
* Github Org - https://github.com/clj-python/
* Carin's NextJournal Page - https://nextjournal.com/gigasquid/
* Kommen's NextJournal Page - https://nextjournal.com/kommen/

Enjoy!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJvCXRuBnGwqxJ7_jzfXrQqhxGAwQgkXkFWdoKMio0kZ1Q%40mail.gmail.com.

Re: interp2 for Clojure

[Chris Nuernberger]

Hey Charles,

Maybe check out the apache commons-math3 interpolators:
https://commons.apache.org/proper/commons-math/javadocs/api-3.6/org/apache/commons/math3/analysis/interpolation/package-summary.html

On Tue, Jan 14, 2020 at 11:04 AM Charles Harvey III <
charlesnhar...@gmail.com> wrote:

> Greetings.
> A current project I am working on needs to run a Bilinear Interpolation. R
> and Python (SciPy) have functions for this: interp2. I think this is just a
> fancy name for particular matrix multiplication. But I'm not really sure.
> Can I cobble this together in core.matrix? Incanter? Neanderthal? I know
> the question is a little broad, I'm just at the beginning of this.
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/d1ec49f4-b13c-4de0-b2f2-05de343881ec%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJvGtA3TmTfZeLDFGOLxu_1Kr_UaRrc-Q6A8cd1Xa311DA%40mail.gmail.com.

Re: Can't use my macro in fuction definition

[Chris Nuernberger]

Hey Ru,

Renaming x to anything will result in roughly the same error in your
function.  The problem is that your instance macro needs to know the
classname at compile time.  As x is a runtime variable, the compiler cannot
see the literal value of x at compile time.  Put another way, in your test
function, with no other changes, you need a switch statement that tests if
x is a known type and then calls the macro with the
now-known-at-compile-time literal classname.


An instance function that found the constructor of x via reflection and
then called that is what you want here.


Chris

On Sat, Oct 19, 2019 at 8:03 AM ru  wrote:

> Ok,  Matching Socks.
>
> On what name should I replace the variable name "x" in the function
> definition so that the macro like it?
>
> суббота, 19 октября 2019 г., 14:09:01 UTC+3 пользователь Matching Socks
> написал:
>>
>> The macro is a code generator, with which the compiler computes the
>> actual definition of the "test" function.  What's there is x, and the macro
>> does not like x.  The REPL definition of JLabel worked because JLabel was
>> the literal argument.  See https://clojure.org/reference/macros, which
>> is subtle about this point, or the excellent book "Mastering Clojure
>> Macros".
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/4bdd8dee-0b87-42a1-8d3a-4a2cf60f3aff%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJsgb2Z%3D9gVE%3DUcrfRiQTtHoqXDDein_tg9J%2B9EXcpekNw%40mail.gmail.com.

[ANN] Blog post about Clojure and Python Interop

[Chris Nuernberger]

We published a blog post about one specific but important tactic I used
when writing libpython-clj in order to allow python to call clojure
functions. I don't think this is necessary knowledge to use panthera or
libpython-clj but it is a key piece to building a new binding to a C system
elsewhere and just a good general piece of knowledge about how certain
things work in the C world:

http://techascent.com/blog/functions-across-languages.html


Also, libpython-clj, the underlying tech stack, and Alan Marazzi's Panthera
all have new versions out.

Enjoy :-)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJtLhd%2BNwhUBmUL3A%3Di%2BHbvTNf3%2BcHAychsbN17Rry4_Sw%40mail.gmail.com.

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

[Chris Nuernberger]

Thad,

You approach seems very promising to me for a lot of jobs.  Spark runs on
top of many things.

As far as a clojure layer on top, what do you think about sparkling
?

On Thu, Jul 4, 2019 at 8:43 AM Thad Guidry  wrote:

> "Batch" - doing things in chunks
> "Processing" - THE WORLD :-)  because it means so many different things to
> so many folks (including your boss)
>
> Without a doubt, you will love Apache Spark for your batch processing and
> writing Spark Programs to conquer any World you are building.
> Spend time to install Spark standalone deploy and then use its powerful
> Spark Shell  (the
> feeling of Clojure REPL  !!)
> If you just want to jump in to a public cluster and Try Spark, then I
> would suggest Databricks .
> Spend time reading the features under Libraries drop-down menu on Apache
> Spark website .
>
> You might even be encouraged enough to write an official API in Clojure
> for Apache Spark within a year!  (win-win)
>
> One note of caution if you are building something for long term, you will
> eventually have a need for data versioning, ACID transactions, schema
> evolution, for this I use Delta Lake  (not Datomic)
> since its fully compatible with Spark
>
> Best of luck!
> Thad
> https://www.linkedin.com/in/thadguidry/
>
>
> On Thu, Jul 4, 2019 at 3:22 AM orazio  wrote:
>
>> Hi @atdixon and Thad, thanks for your help.
>>
>> I provide more details about my project
>> My big data layer  is inspired by Lambda architecture. The pipeline
>> include following layers and related tool choosed to address the issue:
>> - *Nifi* for *data ingestion*, and publisinh data/message on  kafka
>> topic.
>> - *Kafka* as *message broker* that with kafka connect, allow me to store
>> data in mongodb ( with mongodb sink and 1 day retention period ) and HDFS
>> (hdfk sink with 1 year retention period)
>> - *Real time processing* with *mongoDB* using it's built-in QueryEngine
>> taht provides extensive Querying, Filtering, and Searching abilities.
>> - *Batch processing* of data stored on HDFS, that performs data
>> aggregation and store result on a HBase Table. *?* The question is :
>> Which tool do you suggest to use for data processing sotred on HDFS ?
>> - *Serving Layer* with *HBase/Phoneix* to store and allow access to
>> batch view.
>>
>> Now i'm invoking your help to choose *the most appropriate tool to
>> execute batch jobs (map reduce)* which will have to aggregate data.
>> Natahn Marz suggests Clojure/Cascalog. Do you know other excellent
>> clojure/Hadoop work in the community, about data processing?
>> if you know some particularly appropriate tools, I could also consider
>> other work/library outside the clojure community.
>>
>> Thanks
>>
>>
>>
>> Il giorno mercoledì 3 luglio 2019 14:56:09 UTC+2, Thad Guidry ha scritto:
>>>
>>> "The best code is never written"
>>>
>>> https://zeppelin.apache.org/
>>> https://nifi.apache.org/
>>>
>>> Thad
>>> https://www.linkedin.com/in/thadguidry/
>>>
>>>
>>> On Tue, Jul 2, 2019 at 11:07 AM orazio  wrote:
>>>
 Hi All,

 I'm newbie on Clojure/Big Data, and i'm starting with hadoop.
 I have installed Hortonworks HDP 3.1
 I have to design a Big Data Layer that ingests large iot datasets and
 social media datasets, process data with MapReduce job and produce
 aggregation to store on HBASE tables.

 For now, my focus is addressed on data processing issue. My question
 is: Is Clojure a good choice for distributed data processing on hadoop ?
 I found Cascalog as fully-featured data processing and querying library
 for Clojure or Java. But are there any active maintainers, for this library
 ?
 Do you know other excellent clojure/Hadoop work in the community,
 abaout data processing?

 I would appreciate some help.

 Orazio

 --
 You received this message because you are subscribed to the Google
 Groups "Clojure" group.
 To post to this group, send email to clo...@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clo...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google
 Groups "Clojure" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to clo...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/clojure/fbc26ffb-5f00-46a7-bf33-7a899f1ffead%40googlegroups.com
 
 .
 For more options, visit 

Re: Java Interop on steroids?

[Chris Nuernberger]

OK that makes more sense, BeamFn is not an interface. I imagine they argued
about that decision at some point.

In your example, MyFn looks to be a general class as it can work with any
clojure var. Do you have a package with many of these type of stub classes
defined? I mean, you could name it BeamFnIntegerString thus encoding the
specializations of the baseclass into the type. One of my questions was
about the range of types that end up specializing the BeamFn<>.

Great link about the proxy attack, that is very interesting.  I want to
dive into that one a bit more.

On Tue, Jul 2, 2019 at 8:14 PM atdixon  wrote:

> I'm glad someone else is thinking on this too!
>
> #2 - For my case at the moment (Apache Beam), I believe we will always
> know the types in advance so using a Java class is workable but of course a
> (proxy++) would be ideal. Beam asks for us to extend abstract generic class
> so we must use (proxy). It also asks for our instances to be Serializable
> (again, proxy explicitly refuses to help here
> <https://clojure.atlassian.net/browse/CLJ-2204> but I believe this, too,
> should be surmountable, without the security implications of that link). In
> any case, what we do looks like this:
>
> // pseudo-ish code
> public class MyFn extends BeamFn implements Serializable
> {
>public MyFn(Var fn) { this.fn = fn; }
>@Override public String invoke(Integer input) { return (String) fn.
> invoke(input); }
> }
>
> On the Clojure side:
>
> ;; pseudo-ish code
>
> (defn my-fn [^Integer val] ...return a string...)
>
> ...
>(register-beam-step (MyFn. #'my-fn))
> ...
>
> Note how we pass the Clojure function as a Var; this is b/c Beam wants to
> send the function over the wire. IFn is not serializable; Var is however
> and can be resolved back to Clojure function on the other end during
> deserialization.
>
> Now of course this whole dance could be eliminated with a specialized
> library that included a proxy++ function that included the ability for
> clients to specify generic type parameter values and serialization support
> (the mechanics of which would need ironing out but I think should be
> possible.)
>
> And to your point about types not known at runtime... this proxy would
> support that use case on-the-fly, which opens a bunch of possible
> interesting options, as well.
>
>
> On Tuesday, July 2, 2019 at 8:24:41 PM UTC-5, Chris Nuernberger wrote:
>>
>> eglue,
>>
>> 1.  I think this is a great idea if it is really necessary.  I would be
>> in favor of a reify++ alone to simplify things.  I find reify amazing at
>> code compression and heavily use it via type specific macros to implement
>> interfaces that for instance support a particular primitive type.
>> 2.  Is a possible workaround to define java interfaces that implement the
>> type specific generic interfaces and then reify those explicitly or is the
>> set of possible interface specialization types unknown a-priori?
>> 3.  The case where something is unbounded or unknown a-priori I would
>> think would often end up with a java class as on of the specializations.
>> In this case, regardless of the cause, one answer might be an upgraded
>> reify pathway.
>> 4.  Are these perhaps cases where you can create just a little bit of
>> java as a generator somehow to generate the interface you need to reify?
>>
>>
>> I would personally find reify a much nicer pathway than calling clojure
>> vars from java.
>>
>> I also think there could be low hanging fruit (or just good unknown
>> libraries) in the pathway for calling clojure vars from java.
>>
>>
>> Interesting problem (at least to me), thanks!
>>
>> On Thu, Jun 20, 2019 at 10:03 PM eglue  wrote:
>>
>>> Don't get me wrong, I'm as much against types as the next R̶i̶c̶h̶
>>> ̶H̶i̶c̶k̶e̶y̶  guy.
>>>
>>> However -- there are many popular Java frameworks that love to reflect
>>> on their annotations and their generic type signatures.
>>>
>>> To name a heavyweight: Spring. But also, of late: big data frameworks,
>>> many written in Java, love reflecting on generic type signatures. My org is
>>> looking at Beam and Flink, for example.
>>>
>>> These frameworks use types not for the static checking really but as
>>> parameters governing their own dynamic behavior. For example, Spring will
>>> use types at runtime to simply match objects to where they should be
>>> dynamically injected. Beam will look at your type signatures and do runtime
>>> validations to ensure it can process things appropriately. Of course this
>>> is unfortunate, us

Re: Java Interop on steroids?

[Chris Nuernberger]

5.  If you need a concrete class definition that then implements a set of
type specific interfaces this would seem to fall into a category of
gen-class assuming you could specify the interfaces with type
specifications.  I can't immediately place a way to do this with anything
mentioned above.  It may be then important to be able to override Object
methods but maybe you already can with gen-class.

On Tue, Jul 2, 2019 at 7:24 PM Chris Nuernberger 
wrote:

> eglue,
>
> 1.  I think this is a great idea if it is really necessary.  I would be in
> favor of a reify++ alone to simplify things.  I find reify amazing at code
> compression and heavily use it via type specific macros to implement
> interfaces that for instance support a particular primitive type.
> 2.  Is a possible workaround to define java interfaces that implement the
> type specific generic interfaces and then reify those explicitly or is the
> set of possible interface specialization types unknown a-priori?
> 3.  The case where something is unbounded or unknown a-priori I would
> think would often end up with a java class as on of the specializations.
> In this case, regardless of the cause, one answer might be an upgraded
> reify pathway.
> 4.  Are these perhaps cases where you can create just a little bit of java
> as a generator somehow to generate the interface you need to reify?
>
>
> I would personally find reify a much nicer pathway than calling clojure
> vars from java.
>
> I also think there could be low hanging fruit (or just good unknown
> libraries) in the pathway for calling clojure vars from java.
>
>
> Interesting problem (at least to me), thanks!
>
> On Thu, Jun 20, 2019 at 10:03 PM eglue  wrote:
>
>> Don't get me wrong, I'm as much against types as the next R̶i̶c̶h̶
>> ̶H̶i̶c̶k̶e̶y̶  guy.
>>
>> However -- there are many popular Java frameworks that love to reflect on
>> their annotations and their generic type signatures.
>>
>> To name a heavyweight: Spring. But also, of late: big data frameworks,
>> many written in Java, love reflecting on generic type signatures. My org is
>> looking at Beam and Flink, for example.
>>
>> These frameworks use types not for the static checking really but as
>> parameters governing their own dynamic behavior. For example, Spring will
>> use types at runtime to simply match objects to where they should be
>> dynamically injected. Beam will look at your type signatures and do runtime
>> validations to ensure it can process things appropriately. Of course this
>> is unfortunate, using types this way, when it is all really just data.
>> Clojure does -- or would do -- it better, simpler, directer, and all of
>> that.
>>
>> Yet we would like to leverage these frameworks. Or rather, we must for
>> various pragmatic and business reasons.
>>
>> And any time we need to "communicate" to these frameworks "through" their
>> desired fashion of generic types and annotations, we can, of course, create
>> the appropriate .java files to represent what is needed (and then do the
>> invocation back to Clojure via IFn.invoke or Compiler.eval, etc). Yes, this
>> works.
>>
>> However this is quite tedious because in these frameworks I mentioned you
>> end up having to create these Java files quite a bit. For example, when
>> expressing a streaming data pipeline to Beam, you may specify multiple
>> transforms, each a function with its own type signature.
>>
>> A little searching and it seems Clojure has shied away from generating
>> generic type information in places where it could offer this capability.
>>
>> For example, in `proxy` ... or I suppose also in `gen-class`, `reify`,
>> and other dynamic bytecode generation features of Clojure.
>>
>> However it seems to me that `proxy` (and these others) could allow one to
>> pass in a representation of desired type arguments, annotations, etc. and
>> then we could remain in Clojure to interop with these popular frameworks.
>>
>> I respect Clojure's efforts to keep its core small and wait for worthy
>> features to prove themselves.
>>
>> So my question is not when is Clojure going to do this, but rather:
>>
>> Are there any precedents in the community for someone building out the
>> kind of richer Java interop that I'm nodding toward here?
>>
>> For example, does anyone know of an attempt out there to build a `proxy`
>> plus-plus, that would allow one to extend a generic class with provided
>> type parameters and have this metadata properly rendered in the bytecode
>> that proxy produces?
>>
>> If not, as a practical and hopefully q

Re: Java Interop on steroids?

[Chris Nuernberger]

eglue,

1.  I think this is a great idea if it is really necessary.  I would be in
favor of a reify++ alone to simplify things.  I find reify amazing at code
compression and heavily use it via type specific macros to implement
interfaces that for instance support a particular primitive type.
2.  Is a possible workaround to define java interfaces that implement the
type specific generic interfaces and then reify those explicitly or is the
set of possible interface specialization types unknown a-priori?
3.  The case where something is unbounded or unknown a-priori I would think
would often end up with a java class as on of the specializations.  In this
case, regardless of the cause, one answer might be an upgraded reify
pathway.
4.  Are these perhaps cases where you can create just a little bit of java
as a generator somehow to generate the interface you need to reify?


I would personally find reify a much nicer pathway than calling clojure
vars from java.

I also think there could be low hanging fruit (or just good unknown
libraries) in the pathway for calling clojure vars from java.


Interesting problem (at least to me), thanks!

On Thu, Jun 20, 2019 at 10:03 PM eglue  wrote:

> Don't get me wrong, I'm as much against types as the next R̶i̶c̶h̶
> ̶H̶i̶c̶k̶e̶y̶  guy.
>
> However -- there are many popular Java frameworks that love to reflect on
> their annotations and their generic type signatures.
>
> To name a heavyweight: Spring. But also, of late: big data frameworks,
> many written in Java, love reflecting on generic type signatures. My org is
> looking at Beam and Flink, for example.
>
> These frameworks use types not for the static checking really but as
> parameters governing their own dynamic behavior. For example, Spring will
> use types at runtime to simply match objects to where they should be
> dynamically injected. Beam will look at your type signatures and do runtime
> validations to ensure it can process things appropriately. Of course this
> is unfortunate, using types this way, when it is all really just data.
> Clojure does -- or would do -- it better, simpler, directer, and all of
> that.
>
> Yet we would like to leverage these frameworks. Or rather, we must for
> various pragmatic and business reasons.
>
> And any time we need to "communicate" to these frameworks "through" their
> desired fashion of generic types and annotations, we can, of course, create
> the appropriate .java files to represent what is needed (and then do the
> invocation back to Clojure via IFn.invoke or Compiler.eval, etc). Yes, this
> works.
>
> However this is quite tedious because in these frameworks I mentioned you
> end up having to create these Java files quite a bit. For example, when
> expressing a streaming data pipeline to Beam, you may specify multiple
> transforms, each a function with its own type signature.
>
> A little searching and it seems Clojure has shied away from generating
> generic type information in places where it could offer this capability.
>
> For example, in `proxy` ... or I suppose also in `gen-class`, `reify`, and
> other dynamic bytecode generation features of Clojure.
>
> However it seems to me that `proxy` (and these others) could allow one to
> pass in a representation of desired type arguments, annotations, etc. and
> then we could remain in Clojure to interop with these popular frameworks.
>
> I respect Clojure's efforts to keep its core small and wait for worthy
> features to prove themselves.
>
> So my question is not when is Clojure going to do this, but rather:
>
> Are there any precedents in the community for someone building out the
> kind of richer Java interop that I'm nodding toward here?
>
> For example, does anyone know of an attempt out there to build a `proxy`
> plus-plus, that would allow one to extend a generic class with provided
> type parameters and have this metadata properly rendered in the bytecode
> that proxy produces?
>
> If not, as a practical and hopefully quick and workable solution, I was
> thinking it'd be possible to take the bytecode emitted by proxy and re-run
> it through ASM to create a *new* class with simply the proxy-produced class
> bytes filled-in with the desired, provided type parameters. I bet this
> could be sufficient and fast, with the slight overhead of the extra class.
>
> To do this, I think I'd need access to these proxy-made bytes... either by
> having proxy answer them somehow, or offering a hook to contribute to the
> defined bytecode before it is committed to the classloader, or by having
> DynamicClassLoader have these bytes on hand for inquiring parties, or
> something else along these lines. This would likely be something that
> Clojure core would have to expose .. correct me if I'm wrong.
>
> Would love to hear any other immediate thoughts on this.
>
> I think once you realize that this generic type information is not even
> being used for "static typing" by these frameworks but rather as an (albeit
> poor) means to 

Re: Java Interop on steroids?

[Chris Nuernberger]

Nathan,

For the subset of problems for which clojure has insufficient power to 
optimize well, I think a good percentage of those would be best off solved 
by numeric languages like numpy <https://github.com/cnuernber/libpython-clj> 
or a numeric compiler like TVM <https://github.com/techascent/tvm-clj/> as 
this gives you a pathway to much more scalability and performance in the 
end.  In short, they aren't best solved by Java either but best solved via 
translation to, execution in, and then translation back from a numeric 
form.  Aided by 4 years of understanding the problem this sort of 
transformation should definitely be possible.
​
As for being able to contribute without knowing clojure I think this will 
probably overall lead to an ossification of design over time but the larger 
set of involved people is definitely a great thing in and of itself.  
Potentially the audience for Storm is mainly other jvm languages.

The best person I found who learned clojure fast was already very good at 
python and used Koans <http://clojurekoans.com/>. I think it should be 
reasonable to stay within some of the extent of the language described in 
things like the Koans for the majority of the codebase and still provide a 
pathway for new people.

That's a lot of thinking for a weekend :-).  Thanks for the post it was 
interesting!

Chris
​

On Friday, June 21, 2019 at 5:18:06 PM UTC-6, Nathan Fisher wrote:
>
> Storm recently moved away from Clojure in its core.
>
> https://storm.apache.org/2019/05/30/storm200-released.html
>
> I wonder how much of the legacy Clojure core could be optimised or if they 
> reached an upper limit imposed by the runtime/architecture. That being said 
> I suspect for 90% of orgs they'll never hit that boundary.
>
> On Fri, Jun 21, 2019 at 16:40, Chris Nuernberger  > wrote:
>
>> Sean, 
>>
>> That is an interesting blog post.  Sorry if I am not following everything 
>> but why not use the annotation support in gen-class for those types of 
>> things?
>>
>>
>> https://github.com/clojure/clojure/blob/8af7e9a92570eb28c58b15481ae9c271d891c028/test/clojure/test_clojure/genclass/examples.clj#L34
>>
>> On Friday, June 21, 2019 at 1:29:56 PM UTC-6, Sean Corfield wrote:
>>>
>>> You might be interested in how we provide type-based annotations on 
>>> Clojure functions so that tooling (in our case New Relic) sees those 
>>> annotations:
>>>
>>>  
>>>
>>>
>>> https://corfield.org/blog/2013/05/01/instrumenting-clojure-for-new-relic-monitoring/
>>>
>>>  
>>>
>>> I agree that this could be a lot easier.
>>>
>>>  
>>>
>>> Sean Corfield -- (904) 302-SEAN
>>> An Architect's View -- http://corfield.org/
>>>
>>> "If you're not annoying somebody, you're not really alive."
>>> -- Margaret Atwood
>>>
>>>  
>>> --
>>> *From:* clo...@googlegroups.com  on behalf of 
>>> eglue 
>>> *Sent:* Thursday, June 20, 2019 9:03:45 PM
>>> *To:* Clojure
>>> *Subject:* Java Interop on steroids? 
>>>  
>>> Don't get me wrong, I'm as much against types as the next R̶i̶c̶h̶ 
>>> ̶H̶i̶c̶k̶e̶y̶  guy. 
>>>
>>> However -- there are many popular Java frameworks that love to reflect 
>>> on their annotations and their generic type signatures.
>>>
>>> To name a heavyweight: Spring. But also, of late: big data frameworks, 
>>> many written in Java, love reflecting on generic type signatures. My org is 
>>> looking at Beam and Flink, for example.
>>>
>>> These frameworks use types not for the static checking really but as 
>>> parameters governing their own dynamic behavior. For example, Spring will 
>>> use types at runtime to simply match objects to where they should be 
>>> dynamically injected. Beam will look at your type signatures and do runtime 
>>> validations to ensure it can process things appropriately. Of course this 
>>> is unfortunate, using types this way, when it is all really just data. 
>>> Clojure does -- or would do -- it better, simpler, directer, and all of 
>>> that.
>>>
>>> Yet we would like to leverage these frameworks. Or rather, we must for 
>>> various pragmatic and business reasons.
>>>
>>> And any time we need to "communicate" to these frameworks "through" 
>>> their desired fashion of generic types and annotations, we can, of course, 
>>> create the appropriate .java files to represent what is needed (and then do 
>>> the invocation back 

Re: Java Interop on steroids?

[Chris Nuernberger]

Sean, 

That is an interesting blog post.  Sorry if I am not following everything 
but why not use the annotation support in gen-class for those types of 
things?

https://github.com/clojure/clojure/blob/8af7e9a92570eb28c58b15481ae9c271d891c028/test/clojure/test_clojure/genclass/examples.clj#L34

On Friday, June 21, 2019 at 1:29:56 PM UTC-6, Sean Corfield wrote:
>
> You might be interested in how we provide type-based annotations on 
> Clojure functions so that tooling (in our case New Relic) sees those 
> annotations:
>
>  
>
>
> https://corfield.org/blog/2013/05/01/instrumenting-clojure-for-new-relic-monitoring/
>
>  
>
> I agree that this could be a lot easier.
>
>  
>
> Sean Corfield -- (904) 302-SEAN
> An Architect's View -- http://corfield.org/
>
> "If you're not annoying somebody, you're not really alive."
> -- Margaret Atwood
>
>  
> --
> *From:* clo...@googlegroups.com   > on behalf of eglue >
> *Sent:* Thursday, June 20, 2019 9:03:45 PM
> *To:* Clojure
> *Subject:* Java Interop on steroids? 
>  
> Don't get me wrong, I'm as much against types as the next R̶i̶c̶h̶ 
> ̶H̶i̶c̶k̶e̶y̶  guy. 
>
> However -- there are many popular Java frameworks that love to reflect on 
> their annotations and their generic type signatures.
>
> To name a heavyweight: Spring. But also, of late: big data frameworks, 
> many written in Java, love reflecting on generic type signatures. My org is 
> looking at Beam and Flink, for example.
>
> These frameworks use types not for the static checking really but as 
> parameters governing their own dynamic behavior. For example, Spring will 
> use types at runtime to simply match objects to where they should be 
> dynamically injected. Beam will look at your type signatures and do runtime 
> validations to ensure it can process things appropriately. Of course this 
> is unfortunate, using types this way, when it is all really just data. 
> Clojure does -- or would do -- it better, simpler, directer, and all of 
> that.
>
> Yet we would like to leverage these frameworks. Or rather, we must for 
> various pragmatic and business reasons.
>
> And any time we need to "communicate" to these frameworks "through" their 
> desired fashion of generic types and annotations, we can, of course, create 
> the appropriate .java files to represent what is needed (and then do the 
> invocation back to Clojure via IFn.invoke or Compiler.eval, etc). Yes, this 
> works.
>
> However this is quite tedious because in these frameworks I mentioned you 
> end up having to create these Java files quite a bit. For example, when 
> expressing a streaming data pipeline to Beam, you may specify multiple 
> transforms, each a function with its own type signature.
>
> A little searching and it seems Clojure has shied away from generating 
> generic type information in places where it could offer this capability. 
>
> For example, in `proxy` ... or I suppose also in `gen-class`, `reify`, and 
> other dynamic bytecode generation features of Clojure.
>
> However it seems to me that `proxy` (and these others) could allow one to 
> pass in a representation of desired type arguments, annotations, etc. and 
> then we could remain in Clojure to interop with these popular frameworks.
>
> I respect Clojure's efforts to keep its core small and wait for worthy 
> features to prove themselves.
>
> So my question is not when is Clojure going to do this, but rather:
>
> Are there any precedents in the community for someone building out the 
> kind of richer Java interop that I'm nodding toward here?
>
> For example, does anyone know of an attempt out there to build a `proxy` 
> plus-plus, that would allow one to extend a generic class with provided 
> type parameters and have this metadata properly rendered in the bytecode 
> that proxy produces?
>
> If not, as a practical and hopefully quick and workable solution, I was 
> thinking it'd be possible to take the bytecode emitted by proxy and re-run 
> it through ASM to create a *new* class with simply the proxy-produced class 
> bytes filled-in with the desired, provided type parameters. I bet this 
> could be sufficient and fast, with the slight overhead of the extra class.
>
> To do this, I think I'd need access to these proxy-made bytes... either by 
> having proxy answer them somehow, or offering a hook to contribute to the 
> defined bytecode before it is committed to the classloader, or by having 
> DynamicClassLoader have these bytes on hand for inquiring parties, or 
> something else along these lines. This would likely be something that 
> Clojure core would have to expose .. correct me if I'm wrong.
>
> Would love to hear any other immediate thoughts on this.
>
> I think once you realize that this generic type information is not even 
> being used for "static typing" by these frameworks but rather as an (albeit 
> poor) means to receive semantic information from their clients (as 
> parameters to govern their own dynamic behavior), then 

[ANN] cpython bindings for clojure

[Chris Nuernberger]

Good morning Clojurians,

About 2 months ago during a talk about the tech.ml systems the point was
made that if we could load the python C libraries then that would provide
us with a lot more libraries to draw from.  In addition, I seem to remember
one of the ML talks a concern about lack of libraries being the problem.

So, in the spirit of building bridges and providing clojure with the best
possible libraries for any particular problem on the planet, I took the
last month and built out raw cpython (libpython3.X) bindings based on JNA.

So, today after cleaning things up just a little bit I decided it was time
to announce libpython-clj.  My hope is that there are enough clojuian
pythonistas that we can take this library and make it an absolute joy to
work with; the best python hosting environment that exists.

https://github.com/cnuernber/libpython-clj

Some features include:

* Python dicts are represented in java as java.util.Map's.
* stdout,stderr are overridden to write to *out* and *err*, respectively.
* transparent access to the python interpreter from any thread.
* zero-copy pathways for numpy->tech.datatype tensors and back.  So you can
intermix heavy processing between clojure and python without pay a large
conversion cost.
* Decent performance.  This is built on the raw cpython shared library so
our cost of executing or interoperating with python should not be any
higher than anyone else's.
* two way bridging.  Expose your JVM objects to python and vice versa.

I have a simple keras demo where I use Keras from clojure to train a neural
network included in the examples.  So that proves out keras and tensorflow
access from clojure.

I think this soundly addresses our ability to load cutting edge math and
machine learning libraries.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CADbpEJvOWoxZu0Y_LHi%2B36bEbcOfxCV5n0jd7mhvq8TO9MaD_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Go block starvation issue

[Chris Nuernberger]

Very cool!

If you are at the level where using into with http headers is a hotspot
then you really are getting pretty good perf already.

At the language level have you seen:

https://github.com/lacuna/bifurcan/blob/master/doc/comparison.md

This may help eek a bit more perf out of some specific situation.

On Fri, Apr 12, 2019 at 2:26 PM Ricardo Mayerhofer 
wrote:

> Hi Chris,
> Thank you very much for the thoughtful answer!
>
> We've been profiling CPU usage, most time currently is used in hard to
> optimize operations such as into (which in turn calls reduce) e.g.:
> https://github.com/B2W-BIT/restQL-clojure/blob/f473ddb1dcd30704764bf64b7f88eff27391aa76/src/main/restql/core/runner/request.clj#L215
>
> But we haven't give up and will keep working on it. On the other hand we'd
> like when high CPU usage happens to protect restQL, which means process
> some requests and drop others, keep running.
>
> I don't think the thread pool is used anywhere else. I could only find
> reference of clojure.core.async.impl.dispatch (who stores the pool) in go,
> take and put.
>
> I think the timed-go will help, gonna try it.
>
> Em sexta-feira, 12 de abril de 2019 16:30:21 UTC-3, Chris Nuernberger
> escreveu:
>>
>> Given the nature of the problem I would wonder why the cpu usage is high.
>> restQL seems to be mostly a passthrough entity completely dependent upon IO
>> so there may be something spinning or a pmap somewhere there should not be.
>>
>>- jstacktrace may offer insight here.
>>
>> If nothing is out of whack in terms of what the machine is actually doing
>> then you could implement a timed go macro that skipped its internals if it
>> took more than Y seconds to execute.
>>
>> (defmacro timed-go
>>   [timeout & args]
>>   `(let [time# (get-time)]
>>  (go
>>(when (< (- (get-time) time#) ~timeout)
>>   ~@args
>>
>> I wouldn’t start there though.  Here are some things I would check before
>> I implemented the above macro.
>>
>>- Potentially something is using the same execution pool that
>>clojure’s async library uses for go. This would also cause go starvation.
>>- Or, something go calls is blocking and go can’t see it because of
>>function abstraction (to my knowledge go doesn’t trace into function 
>> calls)
>>or blocking on something other than a channel.
>>- Potentially something in the go pathway is doing too much cpu
>>work.  Move heavy computation *out* of the go pathway itself to a separate
>>executor, potentially lowering the thread priority.   Use the go executor
>>for blocking io only.  Somewhere in there is a computational bandwidth vs.
>>latency tradeoff.
>>
>> Hope this is solved and you can enjoy your weekend!
>>
>> Chris
>>
>>
>>
>>
>> On Fri, Apr 12, 2019 at 12:26 PM Ricardo Mayerhofer 
>> wrote:
>>
>>> We're investigating a starvation issue in restQL (
>>> https://github.com/B2W-BIT/restQL-http) under heavy load. We make heavy
>>> use of core.async, all I/O is non-blocking, we use Aleph as http server and
>>> client.
>>>
>>> The symptom is that it takes a long time for the first line of go
>>> routine to execute. e.g.:
>>>
>>> (defn a []
>>>   (f1)
>>>   (f2)
>>>   (f3)
>>>   (go
>>> (f4)
>>>   )
>>> )
>>>
>>> f1, f2, f3 are executed very fast in sequence. f4 takes more than 10
>>> seconds to execute.
>>>
>>> The CPU usage is high, so I guess the core.async thread pool is full so
>>> it takes a long time for the dispatched block to execute in the thread
>>> pool. This causes every request to timeout because no one able is able to
>>> execute under the allowed time.
>>>
>>> So my questions is:
>>> - Is there a way to cancel a go execution block when it's queued for a
>>> certain amount of time, such as max queue time? When the starvation process
>>> start this would cancel the go routines and release resources.
>>>
>>> Or any idea? Thanks in advance.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clo...@googlegroups.co

Re: Go block starvation issue

[Chris Nuernberger]

Given the nature of the problem I would wonder why the cpu usage is high.
restQL seems to be mostly a passthrough entity completely dependent upon IO
so there may be something spinning or a pmap somewhere there should not be.

   - jstacktrace may offer insight here.

If nothing is out of whack in terms of what the machine is actually doing
then you could implement a timed go macro that skipped its internals if it
took more than Y seconds to execute.

(defmacro timed-go
  [timeout & args]
  `(let [time# (get-time)]
 (go
   (when (< (- (get-time) time#) ~timeout)
  ~@args

I wouldn’t start there though.  Here are some things I would check before I
implemented the above macro.

   - Potentially something is using the same execution pool that clojure’s
   async library uses for go. This would also cause go starvation.
   - Or, something go calls is blocking and go can’t see it because of
   function abstraction (to my knowledge go doesn’t trace into function calls)
   or blocking on something other than a channel.
   - Potentially something in the go pathway is doing too much cpu work.
   Move heavy computation *out* of the go pathway itself to a separate
   executor, potentially lowering the thread priority.   Use the go executor
   for blocking io only.  Somewhere in there is a computational bandwidth vs.
   latency tradeoff.

Hope this is solved and you can enjoy your weekend!

Chris




On Fri, Apr 12, 2019 at 12:26 PM Ricardo Mayerhofer 
wrote:

> We're investigating a starvation issue in restQL (
> https://github.com/B2W-BIT/restQL-http) under heavy load. We make heavy
> use of core.async, all I/O is non-blocking, we use Aleph as http server and
> client.
>
> The symptom is that it takes a long time for the first line of go routine
> to execute. e.g.:
>
> (defn a []
>   (f1)
>   (f2)
>   (f3)
>   (go
> (f4)
>   )
> )
>
> f1, f2, f3 are executed very fast in sequence. f4 takes more than 10
> seconds to execute.
>
> The CPU usage is high, so I guess the core.async thread pool is full so it
> takes a long time for the dispatched block to execute in the thread pool.
> This causes every request to timeout because no one able is able to execute
> under the allowed time.
>
> So my questions is:
> - Is there a way to cancel a go execution block when it's queued for a
> certain amount of time, such as max queue time? When the starvation process
> start this would cancel the go routines and release resources.
>
> Or any idea? Thanks in advance.
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] General ML and ETL libraries

[Chris Nuernberger]

Clojurians,


Good morning from (again) snowy Boulder!


Following lots of discussion and interactions with many people around the
clojure and ML worlds, TechAscent has built a foundation with the intention
to allow the average clojurist to do high quality machine learning of the
type they are likely to encounter in their day to day work.


This isn't a deep learning framework; I already tried that in a bespoke
fashion that and I think the mxnet bindings are great.


This is specifically for the use case where you have data coming in from
multiple data sources and you need to do the cleaning, processing, and
feature augmentation before running some set of simple models.  Then
gridsearch across a range of models and go about your business from there.
Think more small to medium sized datomic databases and such.  Everyone has
a little data before they have a lot and I think this scale captures a far
wider range of possible use cases.


The foundation comes in two parts.


The first is the ETL library:

https://github.com/techascent/tech.ml.dataset

This library is a column-store based design sitting on top of tablesaw.
The clojure ml group profiled lots of different libraries and we found that
tablesaw works great.

The ETL language is composed of three sub languages.  First a set-invariant
column selection language.  Second, a minimal functional math language
along the lines of APL or J.  Finally a pipeline concept that allows you to
describe an ETL pipeline in data with the idea that you create the pipeline
and run it on training data and then it records context.  Then during
inference later you just used the saved pipeline from the first operation.

This is the second large ETL system I have worked on; the first was one
named Alteryx.


The next library is a general ML framework:

https://github.com/techascent/tech.ml

The library has  bindings to xgboost, smile, and libsvm.  Libsvm doesn't
get the credit it deserves, btw, as it works extremely well on small-n
problems.  xgboost works well on everything and smile contains lots of
different types of models that may or may not work well depending on the
problem as well as clustering and a lot of other machine-learny type
things.

For this case, my interest wasn't a clear exposition of all the different
things smile can do as it was more just to get a wide enough domain of
different model generators to be effective.  For a more thorough binding to
smile, check out:

https://github.com/generateme/fastmath


I built a clojure version a very involved kaggle problem example using
clojupyter and oz as a proof of concept:


https://github.com/cnuernber/ames-house-prices/blob/master/ames-housing-prices-clojure.md


Enjoy :-).

Complements of the TechAscent Crew & Clojure ML Working Group

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: avoiding casts with aget & friends

[Chris Nuernberger]

That is why I used 'unchecked-long' instead of 'long'.

(unchecked-long (unchecked-byte 5))

Not

(long (byte 5))


On Wed, Jan 30, 2019, 5:55 PM Brian Craft  I haven't tried much. I'm getting the java via clj-java-decompiler.core
> 'decompile' macro.
>
> A long cast does drop the cast (which is really counter-intuitive:
> explicitly invoke 'long', which calls longCast, in order to avoid calling
> longCast).
>
> Amusingly this doesn't reduce the total run-time, though longCast drops
> out of the hotspot list. :-p There must be some other limiting step that
> I'm missing in the profiler.
>
> I'm calling it around 1.2M times, so hopefully that engages the jit.
>
> On Wednesday, January 30, 2019 at 3:39:41 PM UTC-8, Alex Miller wrote:
>>
>> What have you tried? And how are you getting that Java? I would prefer to
>> look at bytecode (via javap) to verify what you're saying.
>>
>> Have you tried an explicit long cast?
>>
>> (aget flat-dict (bit-and 0xff (long (aget arr j
>>
>> Are you running this hot enough for the JIT to kick in? Usually this is
>> the kind of thing it's good at, but it might take 10k invocations before it
>> does.
>>
>>
>> On Wednesday, January 30, 2019 at 4:03:43 PM UTC-6, Brian Craft wrote:
>>>
>>> Profiling is showing a lot of time spent in RT.longCast, in places like
>>> this:
>>>
>>> (aget flat-dict (bit-and 0xff (aget arr j)))
>>>
>>> arr is hinted as ^bytes, and flat-dict as ^objects.
>>>
>>> which compiles to this:
>>>
>>> Object code2 = RT.aget((Object[])flat_dict, RT.intCast(0xFFL &
>>> RT.longCast((Object)RT.aget((byte[])arr2, RT.intCast(k)
>>>
>>> Is there any way to avoid that RT.longCast? There is an aget method in
>>> RT.java that returns a byte, and a longCast for byte, but I suspect the
>>> cast to Object is causing it to hit the longCast for Object, which does a
>>> bunch of reflection.
>>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: avoiding casts with aget & friends

[Chris Nuernberger]

does doing an unchecked cast of the return value of the aget on the byte
array change things?

(defn test-fn
[]
(let [obj-ary (object-array 10)
  byte-data (byte-array (range 10))]
  (doseq [idx (range 10)]
(let [idx (int idx)]
  (aget obj-ary idx
(bit-and 0xFF (unchecked-long (aget byte-data
idx

This disassembled clearer and I think dropped the long cast.

On Wed, Jan 30, 2019 at 3:03 PM Brian Craft  wrote:

> Profiling is showing a lot of time spent in RT.longCast, in places like
> this:
>
> (aget flat-dict (bit-and 0xff (aget arr j)))
>
> arr is hinted as ^bytes, and flat-dict as ^objects.
>
> which compiles to this:
>
> Object code2 = RT.aget((Object[])flat_dict, RT.intCast(0xFFL &
> RT.longCast((Object)RT.aget((byte[])arr2, RT.intCast(k)
>
> Is there any way to avoid that RT.longCast? There is an aget method in
> RT.java that returns a byte, and a longCast for byte, but I suspect the
> cast to Object is causing it to hit the longCast for Object, which does a
> bunch of reflection.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] metasoarous/oz 1.5.1 - clojure dataviz updates (jupyter notebooks, markdown extensions, and static export, oh my!)

[Chris Nuernberger]

Great improvements Christopher!  The markdown and Jupyter support are
exactly the direction I was hoping for and we really appreciate the work.

On Wed, Jan 23, 2019 at 1:26 AM Christopher Small 
wrote:

>
> Greetings
>
>
> I'm excited to announce the release of oz 1.5.1.
>
> https://github.com/metasoarous/oz
>
> Oz is a simple data visualization library built around Vega & Vega-Lite.
>
> In Vega & Vega-Lite, data visualizations are specified as pure data
> descriptions about how to map properties of your data and interactions to
> aesthetics of a visualization.  To find out more about Vega & Oz please
> visit https://github.com/metasoarous/oz.
>
> This release specifically adds some major new features:
>
> * Jupyter notebook support via the Clojupyter & IClojure kernels
> * Export of visualizations and scientific documents to live/interactive
> html files via the `export!` function
> * Load markdown files, with a notation for embedding visualizations as
> code blocks
> * Cljdoc API documentation (https://cljdoc.org/d/metasoarous/oz)
>
> These features, together with those already built into oz (REPL based
> workflow, hiccup support, Reagent components & publishing/sharing API),
> make oz a powerful tool for working with data visualizations and scientific
> documents from within Clojure, no matter the workflow.  I hope you find it
> useful.
>
>
> *## Markdown support*
>
> I'd like to specifically illustrate the markdown support feature, as its
> the one I'm most excited to start using myself, as well as the one which
> demands the most explanation.
>
> How many times have you been working on a simple markdown document, and
> realized you wanted to add a data visualization to illustrate a point?
> What did you have to do to get it done?  My guess is you had fire up
> another tool, like R's `ggplot`, or Python's `matplotlib`, export a static
> figure, and awkwardly embed it into your markdown document, hoping to God
> you don't have to update it, and go through the ordeal again.
>
> With oz, you can simply embed vega-lite or vega visualizations like this:
>
> # Some markown file
>
> A data visualization:
>
> ```edn vega-lite
> {:data {:values [{:a 2 :b 3 :c "T"} {:a 5 :b 2 :c "T"} {:a 7 :b 4 :c
> "Q"} {:a 3 :b 3 :c "Q"}]}
>  :mark :point
>  :width 400
>  :encoding {:x {:field "a"}
> :y {:field "b"}
> :color {:field "c"}}}
> ```
>
> The `load` function parses the markdown file, and uses the `edn vega-lite`
> code block class to determine that the block should be interpreted as a
> Vega-Lite visualization.  The fact that `edn` is one of the classes here
> means that your text editor and other markdown processors (if you push to
> GitHub or whatever) will recognize what kind of data it is and highlight it
> appropriately.  (Thanks to GH users mpcarolin and yogthos for promptly
> updating their markdown processing libraries to allow for the specification
> of multiple code classes.)
>
> Once loaded, the corresponding document can be immediately viewed with the
> `view!` function, exported to a self-contained html file via `export!`, or
> published online with a shareable link via `publish!`.
>
> This notation allows you to embed as either `json` or `yaml` in lieu of
> `edn`, or `vega` in lieu of `vega-lite`.  Moreover, `hiccup` can be
> embedded (possibly with `[:vega ...]` or `[:vega-lite ...]` nodes), for
> when you want more power than Markdown affords, but don't want to resort to
> manually writing html in your beautiful Markdown.
>
>
> *## Jupyter notebook support*
>
> While I personally prefer creating scientific documents from the comfort
> of my favorite text editor & REPL setup, I understand the value of the
> notebook environment.  In fact, my first programming language was
> Mathematica, and there's a part of me that holds warm reverie for the
> model.  Thus, it is with great pleasure that I announce that Oz can now be
> used as a go-to for those who enjoy using these environments and wish to be
> able to create powerful and interactive data visualizations therein.
>
> This feature would not have been possible without GH users mikeyford,
> keesterbrugge, jtcbrule, cgrand.
> Thank you all for your help initiating and piecing together a solution to
> the problem(s)!
>
> For usage details can be found in the README.
>
>
> *## In closing*
>
> I hope that you find this nexus of features and ideas useful, and that you
> help me make Vega, Vega-Lite & Oz a standard part of the Clojure toolkit
> for data science.
>
> Thanks for your time
>
> Christopher Small
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> 

Re: Any way to replace function body?

[Chris Nuernberger]

There has to be.  Trace, profile, and debugging libraries for clojure do it
all the time.  I might check the source for some of those.

On Sat, Jan 19, 2019 at 9:06 AM jason poage  wrote:

> Why would you want to overwrite a function if you need reference to the
> original function? Why not just rename the wrapper function to something
> else?
>
> Or you could use (source function-name) to get the function body and
> essentially rename the function.
>
> Sent from my iPhone
>
> On Jan 19, 2019, at 07:50, Janko Muzykant  wrote:
>
> Hi,
>
> Is there an way to replace body of existing (interned) function with own
> code still being able to call original fn?
> Suppose, I have a function:
>
> (defn fetch-data [arg1 arg2]
>   (db/fetch-data ...))
>
> I would like to intern a slightly modified version of this fn. Something
> like this:
>
> (defn fetch-data [& args]
>   (let [result (apply original-fetch-data args)]
> (transform-result result)))
>
> The problem I see is how to keep the reference to original fetch-data fn
> (here denoted by original-fetch-data),
> so it could be still called in a altered version of fetch-data function.
>
> Best,
> JM.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: using durable-queue, works locally, get :time-out when moving to an EC2

[Chris Nuernberger]

Are you using aot?

We have used that durable queue with 1.8.  In fact, we have a compatibility
later that allows you to move from the durable queue to an Aws queue:
(mostly undocumented)

https://github.com/techascent/tech.queue

Is the problem possibly a difference between your compilation environment
and your deploy env?



On Wed, Jan 16, 2019, 1:29 PM  So, I upgraded to Java 11, and now everything works. So I guess this was a
> version conflict.
>
> Just curious, but is there a way for Factual to make durable-queue to tell
> Leiningen that Java 11 is necessary?
>
>
>
> On Wednesday, January 16, 2019 at 3:17:49 PM UTC-5, lawrence...@gmail.com
> wrote:
>>
>> On the new EC2 instance, running Ubuntu:
>>
>> java -version
>> openjdk version "1.8.0_191"
>> OpenJDK Runtime Environment (build
>> 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
>> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
>>
>> Is it possible this version does not have the checksum signature that
>> durable-queue is looking for?
>>
>> I'm thinking this is some kind of version problem.
>>
>>
>>
>>
>>
>>
>> On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5, lawrence...@gmail.com
>> wrote:
>>>
>>> Sorry, I'm an idiot. The real error was when I called put!
>>>
>>> I don't understand this error:
>>>
>>> INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
>>>   java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
>>>  at durable_queue$checksum.invokeStatic (durable_queue.clj:64)
>>> durable_queue$checksum.invokePrim (durable_queue.clj:-1)
>>> durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314)
>>> durable_queue$queues$reify__6549$slab_BANG___6570.invoke
>>> (durable_queue.clj:702)
>>> durable_queue$queues$reify__6549$fn__6575.invoke
>>> (durable_queue.clj:719)
>>> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717)
>>> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732)
>>> humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke
>>> (core.clj:665)
>>> humongorous_nlp.core$cycle_to_database$fn__13673.invoke
>>> (core.clj:663)
>>> clojure.lang.AFn.run (AFn.java:22)
>>> java.util.concurrent.Executors$RunnableAdapter.call
>>> (Executors.java:511)
>>> java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308)
>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301
>>> (ScheduledThreadPoolExecutor.java:180)
>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
>>> (ScheduledThreadPoolExecutor.java:294)
>>> java.util.concurrent.ThreadPoolExecutor.runWorker
>>> (ThreadPoolExecutor.java:1149)
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run
>>> (ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run (Thread.java:748)
>>>
>>>
>>>
>>>
>>> On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5,
>>> lawrence...@gmail.com wrote:

 I was away from Clojure for a year and I missed it. I am pleased to be
 back. But I've forgotten certain common errors. I feel like this is
 something I used to know but now I've lost the knowledge.

 I'm using Factual's durable-queue to put a step inbetween the import of
 large JSON files, and their writes to the database. This works fine on my
 local MacBook Pro, but when I move to an EC2 instance, I'm instead getting
 time-outs when durable-queue tries to read from the queue.

 At start up the app creates a few of these workers, which run for as
 long as the app is running:

 (defn worker
   [from-topics-to-persistence-queue current-database-connection]
   (slingshot/try+
(loop [message (durable/take! from-topics-to-persistence-queue
 :message 6 :timed-out!)]
  (slingshot/try+
   (log-seq " the message in the work function " message)
   (when (= (type message) durable_queue.Task)
 (advance message current-database-connection))
   (catch Object o
 (log-seq "error in worker function")
 (durable/retry! message)
 (log-seq o)))
  (recur (durable/take! from-topics-to-persistence-queue :message
 6 :timed-out!)))
(catch Object o
  (error o)
  (slingshot/throw+ {
 :type worker
 :error o
 :from-topics-to-persistence-queue
 from-topics-to-persistence-queue
 :current-database-connection
 current-database-connection
 }


 On the EC2 instance, I see in the output:

 Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
 INFO:  the message in the work function

 Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
 INFO:

 Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
 INFO: :timed-out!

 I'm using an 

ANN - tech.resource

[Chris Nuernberger]

Clojurians -

I hope your holidays are going well!

We took the time to explain in some detail a small library that has been
extremely useful since the days of cortex:

blog post:

http://techascent.com/blog/generalized-resource-management.html

Library:

https://github.com/techascent/tech.resource

It enables either stack-based resource management similar to with-open and
gc-based resource management.  We hope you find it useful and that it
simplifies using 'off-heap' resources.


Complements from cold, snowy Boulder!

Chris

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] deps.edn dependency viewer

[Chris Nuernberger]

Newest version - Supports Leiningen and TADA:

1.  Now there is an sh script.  You can just install this and use it
anywhere - no plugins, no muss, no fuss.
2.  More graph manipulation options - highlight, prune, focus, remove.
3.  Leiningen is supported along with profiles.  That being said, I am
highjacking leiningen-core so if you have fancy plugins or something like
that it may fail.  A simple workaround is to create a deps.edn file that
states the dependencies that you really do care about.

Note that leiningen's dependency resolution is fairly opaque to me and thus
I may not choose exactly the same resolution that lein deps :tree will.
Deps.edn will always resolve exactly the way tools.deps will resolve it
aside from completely unsupported options.


I sincerely hope this greatly eases the pain when some apache project fails
to reflect some internal class of jackson-fasterxml because your
dependencies changed in some seemingly unrelated way.  If you haven't been
there yet...you will be.  {scary face and voice} You will be.


https://github.com/cnuernber/depsviz/

On Thu, Dec 20, 2018 at 6:58 AM Alex Miller  wrote:

> #winning
>
> On Thursday, December 20, 2018 at 7:52:48 AM UTC-6, Tim Visher wrote:
>>
>> On Tue, Dec 18, 2018 at 12:46 PM Alex Miller  wrote:
>>
>>> tda, tda.reader, tda.extensions
>>>
>>
>> To make an unrelated, unhelpful, and off-topic comment I will now be
>> referring to `tools.deps.alpha` exclusively as 'Tada!'. :)
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] deps.edn dependency viewer

[Chris Nuernberger]

Clojurians-1.10.0:

I recently released a small project that allows you to view the entire
dependency graph of your deps.edn files.  You can do this from the command
line without installing anything or adding anything to your project.

https://github.com/cnuernber/depsviz


To do this, I had to get into some internal symbols in tools.deps.alpha and
I am relying heavily on its graph representation.  At some point, a
discussion about a public API/ABI about tools.deps.alpha would be good.


This came out of work to bring the excellent vizdeps leiningen plugin up to
date with leiningen 2.8.3:

(https://github.com/clj-commons/vizdeps)

I know that at least we have saved a *ton* of time using vizdeps.  Plus we
have been able to get visibility into larger systems and file appropriate
issues with appropriate projects when they had a serious transitive
dependency problems.

In any case, I hope that some of you find this helpful!  Please feel free
to file issues and PRs if something comes up.  I don't currently support
even nearly the entire range of possibilities in a deps.edn file (aliases
for example) so there are lots of opportunites for discussions and PRs.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Clojure 1.10.0-beta8

[Chris Nuernberger]

Agreed, specifically to avoid things like this:
https://github.com/tech-ascent/tech.compute/blob/master/src/tech/compute/cpu/tensor_math.clj#L45

Great timing and we will be looking forward to it!

On Wed, Nov 21, 2018 at 11:09 AM Sean Corfield  wrote:

> This makes me smile… My first reaction to the name was the same as
> Alexander’s and then my second reaction cut in: “No, they’ll have thought
> about the name and won’t entertain changes” 
>
>
>
> It’s for safely doing requires in asynchronous (multi-threaded) code.
>
>
>
> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
> An Architect's View -- http://corfield.org/
>
> "If you're not annoying somebody, you're not really alive."
> -- Margaret Atwood
>
>
> --
> *From:* clojure@googlegroups.com  on behalf of
> Alex Miller 
> *Sent:* Wednesday, November 21, 2018 9:54:28 AM
> *To:* clojure@googlegroups.com
> *Subject:* Re: [ANN] Clojure 1.10.0-beta8
>
> We’re good with the name. The docstring exists for further explanation.
>
> On Nov 21, 2018, at 11:29 AM, Alexander Yakushev 
> wrote:
>
> Could I suggest bikeshedding on the name async-require? Before I've seen
> the patch, my initial impression was that it loads namespaces
> asynchronously (that is, returns control immediately and loads them in the
> background). It might be somewhat confusing that a function async-require
> is actually even more synchronous than the regular require :). How about 
> synchronized-require
> (it's basically word-to-word description of the function body)?
>
> Otherwise, thanks for this one, I needed something like this plenty of
> times!
>
> On Wednesday, November 21, 2018 at 5:17:26 PM UTC+2, Alex Miller wrote:
>>
>> 1.10.0-beta8 is now available.
>>
>> You can try it with clj using:
>>
>> clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.0-beta8"}}}'
>>
>> Changes in 1.10.0-beta8:
>>
>>- CLJ-2438  - demunge
>>source location in execution error messages
>>- CLJ-2437  - add
>>async-require and use it from requiring-resolve
>>- CLJ-2436  - fix
>>reflection warning in reflect.java
>>
>> You can read the full 1.10 changelog here:
>> https://github.com/clojure/clojure/blob/master/changes.md
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/2GQQpxNcDlM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at

[ANN] High Performance Numerics Compiler

[Chris Nuernberger]

Clojurians, we are happy to put forth our work bringing the TVM compiler
and infrastructure to clojure.  It has backends for ARM, Intel, OpenCV,
Cuda, OpenGL, Vulkan, ROCm, and more.

This system is currently getting state of the art performance several deep
learning kernels and I demonstrate it getting great performance (usually
beating by a sound margin but not always) as compared to hand optimized
code in OpenCV.

Utilizing years of experience writing doing HPC and GPGPU programming, we
researched the best way to integrate clojure at a low level with one of the
major NN toolkits.  We didn't want to sit on top of them alone because we
believe the underlying technology is useful in a broader context and we
believe that Clojure should be a first class citizen in these ecosystems.

So we spent roughly the last 6 months working on and off in infrastructure
and support systems and you saw an example of the result earlier this year
with the opencv bindings.

So, with no further ado:

1.  Light description of the overall compiler architecture and theory:
http://techascent.com/blog/high-performance-compilers.html

2.  Example of using the compiler:
http://techascent.com/blog/tvm-for-the-win.html

3.  Github project (with lots of links to more material):
https://github.com/tech-ascent/tvm-clj

Happy Monday from snowy Boulder, Colorado :-)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

clojure contrib build failure on win7

[chris nuernberger]

Hey all,

I am running a fairly new win7-64 system (jdk6-64) and the maven build
of clojure contrib fails due to test failures in at least logging.

Do you want stack traces and debug logs and such?

Chris

-- 
A foolish consistency is the hobgoblin of little minds - Emerson

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en