Re: clojure success story ... hopefully :-)
On Sat, Aug 22, 2009 at 2:08 AM, Sigrid keyd...@gmx.de wrote: Hi Meikel, hi all, thanks for the explanation, I think I got it now. I suppose something in the sentence I quoted led me to think that pattern matching was less in a way than destructuring, whereas in fact it seems to be the opposite - pattern matching seems to presuppose destructuring if I'm correct now. Correct, pattern matching is built using destructuring bind. Still then (regarding The way that Rich elected to de-couple destructuring bind from pattern matching was brilliant.) , it is unclear to me why it was such a good idea not to include pattern matching, or, to somehow keep them separate... Destructuring is useful all over the place, not just for pattern matching. For example, it is really useful in function parameter vectors. Ciao, Sigrid On 21 Aug., 21:16, Meikel Brandmeyer m...@kotka.de wrote: Hi, Am 21.08.2009 um 20:02 schrieb Sigrid: Could someone point me to what the difference is? I know pattern matching e.g. from the PLT scheme implementation, and there the pattern matching also provides the binding and destructuring I think...? The difference is, that in pattern matching you can also specify values on the left side. For example in OCaml: type foo = [ Foo of int ]; value frobnicate x = match x with [ Foo 5 - do_something () | Foo 7 - do_something_else () | Foo x - do_more x ]; (Please bear with me if I don't remember all the details of the syntax.) While this is not possible in Clojure: (let [[x 5 y] [1 2 3]] ...) The five on the left hand side is not allowed. Hope this helps. Sincerely Meikel smime.p7s 2KAnzeigenHerunterladen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
On Sat, 2009-08-22 at 23:58 -0700, bradford cross wrote: On Sat, Aug 22, 2009 at 11:24 PM, Michel Salim michael.silva...@gmail.com wrote: On Sat, 2009-08-22 at 23:00 -0700, bradford cross wrote: Destructuring is useful all over the place, not just for pattern matching. For example, it is really useful in function parameter vectors. I consider that to be an example of pattern matching, though. As far as I understand it. Pattern matching is built from destructuring bind, but destructuring bind is not pattern matching. Pattern matching follows a match-when-return, or match-with-return logical flow written as: [match_val] - return_val Fair enough; I guess it's just that, prior to Clojure, you tend to get one with the other (thus pattern-matching functional language let you both bind or match, both within a function and (for Haskell, at least) in the function declaration as well. Regards, -- Michel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
Hi Meikel, hi all, thanks for the explanation, I think I got it now. I suppose something in the sentence I quoted led me to think that pattern matching was less in a way than destructuring, whereas in fact it seems to be the opposite - pattern matching seems to presuppose destructuring if I'm correct now. Still then (regarding The way that Rich elected to de-couple destructuring bind from pattern matching was brilliant.) , it is unclear to me why it was such a good idea not to include pattern matching, or, to somehow keep them separate... Ciao, Sigrid On 21 Aug., 21:16, Meikel Brandmeyer m...@kotka.de wrote: Hi, Am 21.08.2009 um 20:02 schrieb Sigrid: Could someone point me to what the difference is? I know pattern matching e.g. from the PLT scheme implementation, and there the pattern matching also provides the binding and destructuring I think...? The difference is, that in pattern matching you can also specify values on the left side. For example in OCaml: type foo = [ Foo of int ]; value frobnicate x = match x with [ Foo 5 - do_something () | Foo 7 - do_something_else () | Foo x - do_more x ]; (Please bear with me if I don't remember all the details of the syntax.) While this is not possible in Clojure: (let [[x 5 y] [1 2 3]] ...) The five on the left hand side is not allowed. Hope this helps. Sincerely Meikel smime.p7s 2KAnzeigenHerunterladen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
There is a slight performance penalty over a normal function call. I think the dispatching takes one function call, a hash lookup, and an equality test. Strictly speaking, an isa? test. That's where the ad hoc hierarchy functionality ties in. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
On Fri, 2009-08-21 at 22:26 -0700, James Sofra wrote: This seems like a pretty nice pattern matching implementation for Clojure. http://www.brool.com/index.php/pattern-matching-in-clojure Beautiful! Cheers, -- Michel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
bradford cross bradford.n.cr...@gmail.com writes: Hi Chad, yep, that was me. We do hope to open source some stuff soon. First will probably be our wrappers for cascading/hadoop and s3. Those would be of great interest to many of us. Please do. --J. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
Hi, I read the related story on InfoQ and found it an extremely interesting and motivating read, Clojure being applied in such an interesting field as machine learning! There is something in the article I'd like to understand better, so i'm just asking here on the group: The way that Rich elected to de-couple destructuring bind from pattern matching was brilliant. Could someone point me to what the difference is? I know pattern matching e.g. from the PLT scheme implementation, and there the pattern matching also provides the binding and destructuring I think...? Excuse me if it's a stupid question, it just made me curious to know :-; Sigrid --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
user= (defmulti length empty?) #'user/length user= (defmethod length true [x] 0) #MultiFn clojure.lang.mult...@1807ca8 user= (defmethod length false [x] (+ 1 (length (rest x #MultiFn clojure.lang.mult...@1807ca8 user= (length [1 2 3 4]) 4 On Fri, Aug 21, 2009 at 12:41 PM, Michel Salimmichael.silva...@gmail.com wrote: On Fri, 2009-08-21 at 11:02 -0700, Sigrid wrote: Hi, I read the related story on InfoQ and found it an extremely interesting and motivating read, Clojure being applied in such an interesting field as machine learning! There is something in the article I'd like to understand better, so i'm just asking here on the group: The way that Rich elected to de-couple destructuring bind from pattern matching was brilliant. Could someone point me to what the difference is? I know pattern matching e.g. from the PLT scheme implementation, and there the pattern matching also provides the binding and destructuring I think...? Clojure allows destructuring of vectors, which happens to be what its functions' argument lists are, so you get most of the benefits of pattern matching. It's not full-blown, though, so (correct me if I'm wrong) the equivalent of this is not possible: length [] = 0 length (_:xs) = 1 + (length xs) Regards, -- Michel -- And what is good, Phaedrus, And what is not good— Need we ask anyone to tell us these things? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
Hi, Am 21.08.2009 um 20:02 schrieb Sigrid: Could someone point me to what the difference is? I know pattern matching e.g. from the PLT scheme implementation, and there the pattern matching also provides the binding and destructuring I think...? The difference is, that in pattern matching you can also specify values on the left side. For example in OCaml: type foo = [ Foo of int ]; value frobnicate x = match x with [ Foo 5 - do_something () | Foo 7 - do_something_else () | Foo x - do_more x ]; (Please bear with me if I don't remember all the details of the syntax.) While this is not possible in Clojure: (let [[x 5 y] [1 2 3]] ...) The five on the left hand side is not allowed. Hope this helps. Sincerely Meikel smime.p7s Description: S/MIME cryptographic signature
Re: clojure success story ... hopefully :-)
On Fri, 2009-08-21 at 12:50 -0700, Kevin Downey wrote: user= (defmulti length empty?) #'user/length user= (defmethod length true [x] 0) #MultiFn clojure.lang.mult...@1807ca8 user= (defmethod length false [x] (+ 1 (length (rest x #MultiFn clojure.lang.mult...@1807ca8 user= (length [1 2 3 4]) 4 Très cool! This could be applied to Meikel's post as well -- you *can* write your own predicate function that in effect test for values. It will just be -- ugly. Is there a performance hit with this style (due to using multimethods) or will this be optimized away in practice? -- Michel signature.asc Description: This is a digitally signed message part
Re: clojure success story ... hopefully :-)
On Aug 21, 5:55 pm, Michel Salim michael.silva...@gmail.com wrote: Is there a performance hit with this style (due to using multimethods) or will this be optimized away in practice? There is a slight performance penalty over a normal function call. I think the dispatching takes one function call, a hash lookup, and an equality test. -SS --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
On Fri, Aug 14, 2009 at 3:10 PM, bradford crossbradford.n.cr...@gmail.com wrote: We have just released flightcaster.com which uses statistical inference and machine learning to predict flight delays in advance of airlines (initial results appear to do so with 85 - 90 % accuracy.) The webserver and webapp are all rails running on the Heroku platform; which also serves our blackberry and iphone apps. The research and data heavy lifting is all in Clojure. Distributed data mining is done via a custom layer on top of cascading (which is a layer on top of hadoop.) All run on EC2 and S3 using the very nice cloudera AMIs and deployment scripts. In addition to the machine learning, the layer atop cascading performs all the complex data data filtering and transformation operations; including distributed joins from heterogeneous data sources and transformations into a time series view that is fed to the machine learning computations that are rolled into mappers and reducers. Remember, this is data from airlines and the FAA, it is not pretty. Web data is messy but we have lots of good frameworks, libs and sanitizers for web data. We wrapped cascading in a thin layer that we use to wrap clojure functions in the cascading function objects and inject those into individual steps in the workflows. This gets us very close to normal function composition for the client code. Ultimately, we want to be able to do normal function composition to compose cascading workflows in the same way as we would would do vanilla function composition for small test runs on our local machines. This is an execution agnostic programming model; client code doesn't bear the signs of distributed execution. As a beneficial side effect, we found that this model forces us to have more fine grained abstractions - because each operation must be ultimately be injectable into a map-reduce phase, otherwise your paralleizm will be unnecessarily course grained. This steers us clear of monolithic uber-expressions. Another aspect of the design that allows us to do this is that the data transformations write out clojure data structure literals, so we are entirely insulated from the normal hadoop input/output formats...the wrapper layer just uses the normal clojure reader to read in the strings from hadoop and apply the vanilla clojure functions to the data structures. But we are not limited to only clojure data structure literals. We also inject other readers that can read other strings to clojure data structures, for example. we use Dan Larkin's wonderful json lib for the initial reads of the raw json data we store. All the analytical code is custom, so we don't use many 3rd party libs outside of cascading, hadoop, the invaluable jets3t for working with s3. Oh, and of course, - since we do so much with temporal analysis - joda-time is the only way to work with dates in a sane way on the jvm. :-) If you travel a lot, check us out: flightcaster.com ... we have iphone and blackberry apps. Unfortunately this is domestic US air travel only at the moment due to the difficulty of of obtaining data for international carriers and aviation agencies. Very cool - congrats! Rich --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
On Aug 14, 2009, at 3:10 PM, bradford cross wrote: We have just released flightcaster.com which uses statistical inference and machine learning to predict flight delays in advance of airlines (initial results appear to do so with 85 - 90 % accuracy.) The webserver and webapp are all rails running on the Heroku platform; which also serves our blackberry and iphone apps. The research and data heavy lifting is all in Clojure. Congratulations to you and your team! I'm glad to see more clojure getting out into production environments. Cheers, - Chas --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
On Fri, Aug 14, 2009 at 7:18 PM, bradford cross bradford.n.cr...@gmail.comwrote: Hi Chad, yep, that was me. We do hope to open source some stuff soon. First will probably be our wrappers for cascading/hadoop and s3. Next might be some core language extensions which might be good in contrib or some other lib. If we release any basic stats or machine learning stuff we may try to merge into incanter if it seems like a fit but haven't had time to check out incanter as I'd like. Very interesting. Are you using (binding [*read-eval* false] ...) when reading Clojure data structures out of strings obtained over your distributed node network? If you're not it's possible you have a security hole that could be exploited by a hostile node masquerading as a legitimate one. (Though likely an attacker would have to penetrate your firewall and get loose in your LAN, gaining privileges on at least one of your machines, to exploit it.) Specifically, a #=() form in the stream would otherwise allow a sort of injection attack. If you use the Clojure reader on other untrusted data, such as fragments of web pages (to parse numbers, say), the same applies: without that binding for those reads, you may be vulnerable in a similar manner. If data from web forms, vulnerable in a very similar manner to SQL injection. Security becomes especially important if you figure to do big parallel reductions on office PC spare cycles instead of dedicated hardware. Those PCs might vary in how sensitive the information on them is, and in how trustworthy their users are. You don't want a newly-hired clerk in sales sending crafted network packets that give him privileges on the desktop computer of the CFO or among the RD department's boxes. The latter lets him sell industrial espionage data to the highest bidder, likely a competitor; the former, possibly do some insider trading or suchlike (and when the SEC shows up to investigate some suspicious trades, they'll be looking at your CFO, as he was the one nominally privy to the inside info). So a breach could cause anything from embarrassment (porn popups during board meeting Powerpoint presentations; intentional pranks) to competitive or legal trouble. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
whoa...missed the google spellcheckers' warning on: paralleizm ... although that may be the proper lolkidde spelling :-) On Fri, Aug 14, 2009 at 12:10 PM, bradford cross bradford.n.cr...@gmail.com wrote: We have just released flightcaster.com which uses statistical inference and machine learning to predict flight delays in advance of airlines (initial results appear to do so with 85 - 90 % accuracy.) The webserver and webapp are all rails running on the Heroku platform; which also serves our blackberry and iphone apps. The research and data heavy lifting is all in Clojure. Distributed data mining is done via a custom layer on top of cascading (which is a layer on top of hadoop.) All run on EC2 and S3 using the very nice cloudera AMIs and deployment scripts. In addition to the machine learning, the layer atop cascading performs all the complex data data filtering and transformation operations; including distributed joins from heterogeneous data sources and transformations into a time series view that is fed to the machine learning computations that are rolled into mappers and reducers. Remember, this is data from airlines and the FAA, it is not pretty. Web data is messy but we have lots of good frameworks, libs and sanitizers for web data. We wrapped cascading in a thin layer that we use to wrap clojure functions in the cascading function objects and inject those into individual steps in the workflows. This gets us very close to normal function composition for the client code. Ultimately, we want to be able to do normal function composition to compose cascading workflows in the same way as we would would do vanilla function composition for small test runs on our local machines. This is an execution agnostic programming model; client code doesn't bear the signs of distributed execution. As a beneficial side effect, we found that this model forces us to have more fine grained abstractions - because each operation must be ultimately be injectable into a map-reduce phase, otherwise your paralleizm will be unnecessarily course grained. This steers us clear of monolithic uber -expressions. Another aspect of the design that allows us to do this is that the data transformations write out clojure data structure literals, so we are entirely insulated from the normal hadoop input/output formats...the wrapper layer just uses the normal clojure reader to read in the strings from hadoop and apply the vanilla clojure functions to the data structures. But we are not limited to only clojure data structure literals. We also inject other readers that can read other strings to clojure data structures, for example. we use Dan Larkin's wonderful jsonlib for the initial reads of the raw json data we store. All the analytical code is custom, so we don't use many 3rd party libs outside of cascading, hadoop, the invaluable jets3t for working with s3. Oh, and of course, - since we do so much with temporal analysis - joda-time is the only way to work with dates in a sane way on the jvm. :-) If you travel a lot, check us out: flightcaster.com ... we have iphone and blackberry apps. Unfortunately this is domestic US air travel only at the moment due to the difficulty of of obtaining data for international carriers and aviation agencies. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
Bradford, I just bought the iPhone app. Looks very cool. I saw a presentation at the JavaOne after-meeting with Rich Hickey about flightcaster. Were you the presenter? The machine learning notation seemed to work very well in Clojure. Are there any portions of this cool stuff that you can share with the community? Chad Harrington chad.harring...@gmail.com On Fri, Aug 14, 2009 at 12:10 PM, bradford cross bradford.n.cr...@gmail.com wrote: We have just released flightcaster.com which uses statistical inference and machine learning to predict flight delays in advance of airlines (initial results appear to do so with 85 - 90 % accuracy.) The webserver and webapp are all rails running on the Heroku platform; which also serves our blackberry and iphone apps. The research and data heavy lifting is all in Clojure. Distributed data mining is done via a custom layer on top of cascading (which is a layer on top of hadoop.) All run on EC2 and S3 using the very nice cloudera AMIs and deployment scripts. In addition to the machine learning, the layer atop cascading performs all the complex data data filtering and transformation operations; including distributed joins from heterogeneous data sources and transformations into a time series view that is fed to the machine learning computations that are rolled into mappers and reducers. Remember, this is data from airlines and the FAA, it is not pretty. Web data is messy but we have lots of good frameworks, libs and sanitizers for web data. We wrapped cascading in a thin layer that we use to wrap clojure functions in the cascading function objects and inject those into individual steps in the workflows. This gets us very close to normal function composition for the client code. Ultimately, we want to be able to do normal function composition to compose cascading workflows in the same way as we would would do vanilla function composition for small test runs on our local machines. This is an execution agnostic programming model; client code doesn't bear the signs of distributed execution. As a beneficial side effect, we found that this model forces us to have more fine grained abstractions - because each operation must be ultimately be injectable into a map-reduce phase, otherwise your paralleizm will be unnecessarily course grained. This steers us clear of monolithic uber -expressions. Another aspect of the design that allows us to do this is that the data transformations write out clojure data structure literals, so we are entirely insulated from the normal hadoop input/output formats...the wrapper layer just uses the normal clojure reader to read in the strings from hadoop and apply the vanilla clojure functions to the data structures. But we are not limited to only clojure data structure literals. We also inject other readers that can read other strings to clojure data structures, for example. we use Dan Larkin's wonderful jsonlib for the initial reads of the raw json data we store. All the analytical code is custom, so we don't use many 3rd party libs outside of cascading, hadoop, the invaluable jets3t for working with s3. Oh, and of course, - since we do so much with temporal analysis - joda-time is the only way to work with dates in a sane way on the jvm. :-) If you travel a lot, check us out: flightcaster.com ... we have iphone and blackberry apps. Unfortunately this is domestic US air travel only at the moment due to the difficulty of of obtaining data for international carriers and aviation agencies. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure success story ... hopefully :-)
Hi Chad, yep, that was me. We do hope to open source some stuff soon. First will probably be our wrappers for cascading/hadoop and s3. Next might be some core language extensions which might be good in contrib or some other lib. If we release any basic stats or machine learning stuff we may try to merge into incanter if it seems like a fit but haven't had time to check out incanter as I'd like. For now this is all on the back burner since building stuff has to be the priority for us and we're people constrained. :) On Aug 14, 2009 4:01 PM, Chad Harrington chad.harring...@gmail.com wrote: Bradford, I just bought the iPhone app. Looks very cool. I saw a presentation at the JavaOne after-meeting with Rich Hickey about flightcaster. Were you the presenter? The machine learning notation seemed to work very well in Clojure. Are there any portions of this cool stuff that you can share with the community? Chad Harrington chad.harring...@gmail.com On Fri, Aug 14, 2009 at 12:10 PM, bradford cross bradford.n.cr...@gmail.com wrote: We have just released flightcaster.com which uses statistical inference and machine learning to ... --~--~-~--~~~---~--~~ You received this message because you are sub... --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
