Re: [julia-users] Re: Environment reification and lazy evaluation
Well a somewhat unique feature or R is automatic argument recycling, so that when doing operations on dataframes, it's a lot easier to incorporate data from outside of the dataframe into evaluation. Let's say you are doing a stepwise logisitic growth model, like dP/dt = r*P*(1 - P/K). Let's say that r a known single value in the global environment, K is going be predicted from some environmental variables within your dataframe, and P is going to be given as an initial condition and its evolution modeled. The R way of looking at this would be to set up an environment where the dataframe environment inherits from the global environment, and to evaluate within this environment regardless of calling environment. That means that within a sub sub function, you can use r to represent, maybe, a row number, and K, to represent, maybe, a column number, and that you can still evaluate an expression like r*P*(1 - P/K) with no problem. This is most useful when writing your own functions. For example, you could build some sort of time-series predictive package that could handle not only logisitic models, but other models, like the Lotka-Volterra models. Now, I'm not saying this is impossible to do in Julia: in fact, I'm quite sure it is. It's just less convenient. The other nice things about environments are more trivial. For example, you might be doing a homework assignment with several problems. At the start of each problem, you could define a new environment. At the end of the problem, you could display the entire content of the problem's environment as a way to track down errors. When looking at an answer sheet, you could compare intermediate results and see where you went wrong. Again, not impossible to do with Julia; you would probably want to set up a dict for each problem, and do all operations within that dict. Which, I guess, is basically what is happening under the hood in a language with environments. I'll just nod in agreement with the fexpr stuff and pretend I understand. On Friday, July 24, 2015 at 4:11:37 PM UTC-4, Scott Jones wrote: I should give you a hard time about arguing from authority ;-), but I was there at the same time and knew KMP, so I wouldn't want to argue too much with him either, and I am in agreement with you about fexprs vs. macros, and the equivalent issues in Julia. On Friday, July 24, 2015 at 2:31:54 PM UTC-4, Stefan Karpinski wrote: Lisp has already been down this path – fexprs https://en.wikipedia.org/wiki/Fexpr were special forms with behavior similar to R's functions, and they were deprecated in the 1980s in favor of macros, https://en.wikipedia.org/wiki/Fexpr#Mainstream_use_and_deprecation: His central objection was that, in a Lisp dialect that allows fexprs, static analysis cannot determine generally whether an operator represents an ordinary function or a fexpr — therefore, static analysis cannot determine whether or not the operands will be evaluated. In particular, the compiler cannot tell whether a subexpression can be safely optimized, since the subexpression might be treated as unevaluated data at run-time. If Lispers think a feature is too dynamic and hard to understand, that's a danger sign. In general, I wouldn't want to argue too much with Kent Pitman :-) On the subject of Kent Pitman, here's a random bit of Lisp wisdom from this interview http://developers.slashdot.org/story/01/11/03/1726251/kent-m-pitman-answers-on-lisp-and-much-more with him: I like Lisp's willingness to represent itself. People often explain this as its ability to represent itself, but I think that's wrong. Most languages are capable of representing themselves, but they simply don't have the will to. Lisp programs are represented by lists and programmers are aware of that. It wouldn't matter if it had been arrays. It does matter that it's program structure that is represented, and not character syntax, but beyond that the choice is pretty arbitrary. It's not important that the representation be the Right® choice. It's just important that it be a common, agreed-upon choice so that there can be a rich community of program-manipulating programs that do trade in this common representation. This is probably the best quote about homoiconicity I've read anywhere and explains why the term is tricky to pin down. On Fri, Jul 24, 2015 at 2:21 PM, Brandon Taylor brandon@gmail.com wrote: Macros seem to have a bunch of limitations. They can't be overloaded to take advantage of Julia's typing system. Their functionality can somewhat mimic evaluation within a function in its direct parent environment, but what if you want to write functions within your macro, or macros inside your macro, etc? Probably still possible to do many things, but it would invovle passing expressions back in forth in confusing ways, where as with environments, as long as an environment is kept attached to every
Re: [julia-users] Re: Environment reification and lazy evaluation
Le samedi 25 juillet 2015 à 01:58 -0700, Brandon Taylor a écrit : Well a somewhat unique feature or R is automatic argument recycling, so that when doing operations on dataframes, it's a lot easier to incorporate data from outside of the dataframe into evaluation. Let's say you are doing a stepwise logisitic growth model, like dP/dt = r*P*(1 - P/K). Let's say that r a known single value in the global environment, K is going be predicted from some environmental variables within your dataframe, and P is going to be given as an initial condition and its evolution modeled. The R way of looking at this would be to set up an environment where the dataframe environment inherits from the global environment, and to evaluate within this environment regardless of calling environment. That means that within a sub sub function, you can use r to represent, maybe, a row number, and K, to represent, maybe, a column number, and that you can still evaluate an expression like r*P*(1 - P/K) with no problem. This is most useful when writing your own functions. For example, you could build some sort of time-series predictive package that could handle not only logisitic models, but other models, like the Lotka -Volterra models. Now, I'm not saying this is impossible to do in Julia: in fact, I'm quite sure it is. It's just less convenient. Thanks for the example. I don't think I would use an environment for that. Why not have the function take a data frame with the variables, a nd constants as an argument? As I understand it, the only interest of environments is that you don't have to make a difference between variables in the data frame and variables in the global scope. But I also find it confusing that you don't know where the data comes from, and I've seen students make mistakes because of that. That said, I don't understand exactly what the code you describe would look like, so I might be missing something. If you can provide a few lines to illustrate a toy example, it could help. The other nice things about environments are more trivial. For example, you might be doing a homework assignment with several problems. At the start of each problem, you could define a new environment. At the end of the problem, you could display the entire content of the problem's environment as a way to track down errors. When looking at an answer sheet, you could compare intermediate results and see where you went wrong. Again, not impossible to do with Julia; you would probably want to set up a dict for each problem, and do all operations within that dict. Which, I guess, is basically what is happening under the hood in a language with environments. You should be able to do this by working inside modules in Julia. In particular, workspace() looks exactly like what you're describing: once you're done with one problem, call it to get a clean environment, storing globals in the LastMain module. Regards I'll just nod in agreement with the fexpr stuff and pretend I understand. On Friday, July 24, 2015 at 4:11:37 PM UTC-4, Scott Jones wrote: I should give you a hard time about arguing from authority ;-), but I was there at the same time and knew KMP, so I wouldn't want to argue too much with him either, and I am in agreement with you about fexprs vs. macros, and the equivalent issues in Julia. On Friday, July 24, 2015 at 2:31:54 PM UTC-4, Stefan Karpinski wrote: Lisp has already been down this path – fexprs were special forms with behavior similar to R's functions, and they were deprecated in the 1980s in favor of macros, https://en.wikipedia.org/wiki/Fe xpr#Mainstream_use_and_deprecation: His central objection was that, in a Lisp dialect that allows fexprs, static analysis cannot determine generally whether an operator represents an ordinary function or a fexpr — therefore, static analysis cannot determine whether or not the operands will be evaluated. In particular, the compiler cannot tell whether a subexpression can be safely optimized, since the subexpression might be treated as unevaluated data at run-time. If Lispers think a feature is too dynamic and hard to understand, that's a danger sign. In general, I wouldn't want to argue too much with Kent Pitman :-) On the subject of Kent Pitman, here's a random bit of Lisp wisdom from this interview with him: I like Lisp's willingness to represent itself. People often explain this as its ability to represent itself, but I think that's wrong. Most languages are capable of representing themselves, but they simply don't have the will to. Lisp programs are represented by lists and programmers are aware of that. It wouldn't matter if it had been arrays. It does matter that it's program structure that is represented, and not character syntax, but beyond that the choice is pretty arbitrary. It's not
Re: [julia-users] Re: Environment reification and lazy evaluation
Cool, workspace() seems very helpful. Yeah, I don't really understand exactly what that code would look like either, because I'm not the most familiar with time-series statistics. My point was that to work easily with flexible models that mix environmental variables with dataframe variables, it is convenient (though again not necessary) to have environment access. On Saturday, July 25, 2015 at 7:34:44 AM UTC-4, Milan Bouchet-Valat wrote: Le samedi 25 juillet 2015 à 01:58 -0700, Brandon Taylor a écrit : Well a somewhat unique feature or R is automatic argument recycling, so that when doing operations on dataframes, it's a lot easier to incorporate data from outside of the dataframe into evaluation. Let's say you are doing a stepwise logisitic growth model, like dP/dt = r*P*(1 - P/K). Let's say that r a known single value in the global environment, K is going be predicted from some environmental variables within your dataframe, and P is going to be given as an initial condition and its evolution modeled. The R way of looking at this would be to set up an environment where the dataframe environment inherits from the global environment, and to evaluate within this environment regardless of calling environment. That means that within a sub sub function, you can use r to represent, maybe, a row number, and K, to represent, maybe, a column number, and that you can still evaluate an expression like r*P*(1 - P/K) with no problem. This is most useful when writing your own functions. For example, you could build some sort of time-series predictive package that could handle not only logisitic models, but other models, like the Lotka -Volterra models. Now, I'm not saying this is impossible to do in Julia: in fact, I'm quite sure it is. It's just less convenient. Thanks for the example. I don't think I would use an environment for that. Why not have the function take a data frame with the variables, a nd constants as an argument? As I understand it, the only interest of environments is that you don't have to make a difference between variables in the data frame and variables in the global scope. But I also find it confusing that you don't know where the data comes from, and I've seen students make mistakes because of that. That said, I don't understand exactly what the code you describe would look like, so I might be missing something. If you can provide a few lines to illustrate a toy example, it could help. The other nice things about environments are more trivial. For example, you might be doing a homework assignment with several problems. At the start of each problem, you could define a new environment. At the end of the problem, you could display the entire content of the problem's environment as a way to track down errors. When looking at an answer sheet, you could compare intermediate results and see where you went wrong. Again, not impossible to do with Julia; you would probably want to set up a dict for each problem, and do all operations within that dict. Which, I guess, is basically what is happening under the hood in a language with environments. You should be able to do this by working inside modules in Julia. In particular, workspace() looks exactly like what you're describing: once you're done with one problem, call it to get a clean environment, storing globals in the LastMain module. Regards I'll just nod in agreement with the fexpr stuff and pretend I understand. On Friday, July 24, 2015 at 4:11:37 PM UTC-4, Scott Jones wrote: I should give you a hard time about arguing from authority ;-), but I was there at the same time and knew KMP, so I wouldn't want to argue too much with him either, and I am in agreement with you about fexprs vs. macros, and the equivalent issues in Julia. On Friday, July 24, 2015 at 2:31:54 PM UTC-4, Stefan Karpinski wrote: Lisp has already been down this path – fexprs were special forms with behavior similar to R's functions, and they were deprecated in the 1980s in favor of macros, https://en.wikipedia.org/wiki/Fe xpr#Mainstream_use_and_deprecation: His central objection was that, in a Lisp dialect that allows fexprs, static analysis cannot determine generally whether an operator represents an ordinary function or a fexpr — therefore, static analysis cannot determine whether or not the operands will be evaluated. In particular, the compiler cannot tell whether a subexpression can be safely optimized, since the subexpression might be treated as unevaluated data at run-time. If Lispers think a feature is too dynamic and hard to understand, that's a danger sign. In general, I wouldn't want to argue too much with Kent Pitman :-) On the subject of Kent Pitman, here's a random bit of Lisp wisdom from this interview
Re: [julia-users] Re: Environment reification and lazy evaluation
Le jeudi 23 juillet 2015 à 15:31 -0700, Brandon Taylor a écrit :
I'm not saying inherent slowness will be particularly useful. I'm
saying environment access will be particularly useful. The point of
Julia is no compromise between speed and code-ability. If it is not
possible to integrate environment access and speed, then I think that
Julia is mis-marketing itself, at least as a viable R alternative.
Could you give us a concrete example of what you would like to do with
environments? I haven't encountered cases where I would personally miss
this R feature in Julia, so I don't understand what you're trying to
achieve.
Also, note that when we say that a solution will be slow in Julia, it
doesn't mean that it would be slower than in R. It's just that to write
code approximately equivalent to C, you cannot use the same solutions
as R.
Regards
On Friday, July 24, 2015 at 12:35:13 AM UTC+4, Yichao Yu wrote:
On Thu, Jul 23, 2015 at 4:23 PM, Brandon Taylor
brandon@gmail.com wrote:
Will it cause a slowdown because there is a better way to do it
or will it
cause a slowdown because something inherent about environment
access?
Because if the second is the case, then it would be worth it, at
least to
me.
AFAICT, environment access is basically global variable and it will
trash any optimization julia can do so in some sense the slow down
is
inherent about environment access.
I don't see why the inherent slowness would be particularly useful
though.
On Thursday, July 23, 2015 at 11:13:14 AM UTC+8, Yichao Yu wrote:
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, made some progress. Still having trouble with a _ENV_ not
defined
error:
how could it not be defined if it's a global variable???
Figuring out
argument passing is going to be tricky. Is this kind of system
feasible
or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the
parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point
it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict
daughter of
the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict
daughter of
the
old environment with dict contents
macro _ADD_ENV_MACRO_(dict)
esc(quote
_DICT_ = $dict
_ADD_ENV_ = gensym()
eval(quote
$_ADD_ENV_ = convert(Dict, $_DICT_)
$_ADD_ENV_[:_parent] = $(Expr(:quote, _ENV_))
end)
_ENV_ = _ADD_ENV_
end)
end
# jump back in time to the previous generation
macro _REMOVE_ENV_MACRO_()
esc(quote
_ENV_ = eval(_ENV_)[:_parent]
end)
end
# new types will have to be included in a module at the
beginning of
code
# that module will need to be converted to a Dict along with
base
# a namespace will need to be created such that module dicts
inherit
from
each other, with base at the top
@_ENV_MACRO_()
@_ADD_ENV_MACRO_(Base)
@_NEW_ENV_MACRO_
# test expression
e =
quote
a = 1
b = 2
test = function()
b = a
end
Re: [julia-users] Re: Environment reification and lazy evaluation
Ok, maybe I was being harsh when I used the word mis-marketing. The post
here: http://julialang.org/blog/2012/02/why-we-created-julia/ specifically
mentions that one of Julia's goals is to be as easy for statistics as
R. Building a domain specific language is more difficult (but clearly not
impossible) without environment access. Again, the examples I'm most
familiar with are in the Hadleyverse (see previous posts in this thread for
links to vignettes). Also, this is probably the fifth time that me being
stubborn seems to have caused tensions on threads, and I'm sorry for that.
Learning a bit about C compilers convinced me that I was completely wrong
about environments needing to be implemented for scoping to work.
On Friday, July 24, 2015 at 3:46:28 AM UTC-4, Milan Bouchet-Valat wrote:
Le jeudi 23 juillet 2015 à 15:31 -0700, Brandon Taylor a écrit :
I'm not saying inherent slowness will be particularly useful. I'm
saying environment access will be particularly useful. The point of
Julia is no compromise between speed and code-ability. If it is not
possible to integrate environment access and speed, then I think that
Julia is mis-marketing itself, at least as a viable R alternative.
Could you give us a concrete example of what you would like to do with
environments? I haven't encountered cases where I would personally miss
this R feature in Julia, so I don't understand what you're trying to
achieve.
Also, note that when we say that a solution will be slow in Julia, it
doesn't mean that it would be slower than in R. It's just that to write
code approximately equivalent to C, you cannot use the same solutions
as R.
Regards
On Friday, July 24, 2015 at 12:35:13 AM UTC+4, Yichao Yu wrote:
On Thu, Jul 23, 2015 at 4:23 PM, Brandon Taylor
brandon@gmail.com wrote:
Will it cause a slowdown because there is a better way to do it
or will it
cause a slowdown because something inherent about environment
access?
Because if the second is the case, then it would be worth it, at
least to
me.
AFAICT, environment access is basically global variable and it will
trash any optimization julia can do so in some sense the slow down
is
inherent about environment access.
I don't see why the inherent slowness would be particularly useful
though.
On Thursday, July 23, 2015 at 11:13:14 AM UTC+8, Yichao Yu wrote:
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, made some progress. Still having trouble with a _ENV_ not
defined
error:
how could it not be defined if it's a global variable???
Figuring out
argument passing is going to be tricky. Is this kind of system
feasible
or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the
parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point
it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict
daughter of
the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict
daughter of
the
old environment with dict contents
macro
Re: [julia-users] Re: Environment reification and lazy evaluation
I can understand that if you're used to building interfaces that use R's
non-standard evaluation and reified environments heavily, it can be
frustrating to not have those features. But I don't think that means that
you cannot build convenient and safe interfaces for data analysis.
On Fri, Jul 24, 2015 at 11:48 AM, Brandon Taylor
brandon.taylor...@gmail.com wrote:
Ok, maybe I was being harsh when I used the word mis-marketing. The post
here: http://julialang.org/blog/2012/02/why-we-created-julia/
specifically mentions that one of Julia's goals is to be as easy for
statistics as R. Building a domain specific language is more difficult (but
clearly not impossible) without environment access. Again, the examples I'm
most familiar with are in the Hadleyverse (see previous posts in this
thread for links to vignettes). Also, this is probably the fifth time that
me being stubborn seems to have caused tensions on threads, and I'm sorry
for that. Learning a bit about C compilers convinced me that I was
completely wrong about environments needing to be implemented for scoping
to work.
On Friday, July 24, 2015 at 3:46:28 AM UTC-4, Milan Bouchet-Valat wrote:
Le jeudi 23 juillet 2015 à 15:31 -0700, Brandon Taylor a écrit :
I'm not saying inherent slowness will be particularly useful. I'm
saying environment access will be particularly useful. The point of
Julia is no compromise between speed and code-ability. If it is not
possible to integrate environment access and speed, then I think that
Julia is mis-marketing itself, at least as a viable R alternative.
Could you give us a concrete example of what you would like to do with
environments? I haven't encountered cases where I would personally miss
this R feature in Julia, so I don't understand what you're trying to
achieve.
Also, note that when we say that a solution will be slow in Julia, it
doesn't mean that it would be slower than in R. It's just that to write
code approximately equivalent to C, you cannot use the same solutions
as R.
Regards
On Friday, July 24, 2015 at 12:35:13 AM UTC+4, Yichao Yu wrote:
On Thu, Jul 23, 2015 at 4:23 PM, Brandon Taylor
brandon@gmail.com wrote:
Will it cause a slowdown because there is a better way to do it
or will it
cause a slowdown because something inherent about environment
access?
Because if the second is the case, then it would be worth it, at
least to
me.
AFAICT, environment access is basically global variable and it will
trash any optimization julia can do so in some sense the slow down
is
inherent about environment access.
I don't see why the inherent slowness would be particularly useful
though.
On Thursday, July 23, 2015 at 11:13:14 AM UTC+8, Yichao Yu wrote:
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, made some progress. Still having trouble with a _ENV_ not
defined
error:
how could it not be defined if it's a global variable???
Figuring out
argument passing is going to be tricky. Is this kind of system
feasible
or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the
parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point
it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict
daughter of
the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote,
Re: [julia-users] Re: Environment reification and lazy evaluation
I should give you a hard time about arguing from authority ;-), but I was there at the same time and knew KMP, so I wouldn't want to argue too much with him either, and I am in agreement with you about fexprs vs. macros, and the equivalent issues in Julia. On Friday, July 24, 2015 at 2:31:54 PM UTC-4, Stefan Karpinski wrote: Lisp has already been down this path – fexprs https://en.wikipedia.org/wiki/Fexpr were special forms with behavior similar to R's functions, and they were deprecated in the 1980s in favor of macros, https://en.wikipedia.org/wiki/Fexpr#Mainstream_use_and_deprecation : His central objection was that, in a Lisp dialect that allows fexprs, static analysis cannot determine generally whether an operator represents an ordinary function or a fexpr — therefore, static analysis cannot determine whether or not the operands will be evaluated. In particular, the compiler cannot tell whether a subexpression can be safely optimized, since the subexpression might be treated as unevaluated data at run-time. If Lispers think a feature is too dynamic and hard to understand, that's a danger sign. In general, I wouldn't want to argue too much with Kent Pitman :-) On the subject of Kent Pitman, here's a random bit of Lisp wisdom from this interview http://developers.slashdot.org/story/01/11/03/1726251/kent-m-pitman-answers-on-lisp-and-much-more with him: I like Lisp's willingness to represent itself. People often explain this as its ability to represent itself, but I think that's wrong. Most languages are capable of representing themselves, but they simply don't have the will to. Lisp programs are represented by lists and programmers are aware of that. It wouldn't matter if it had been arrays. It does matter that it's program structure that is represented, and not character syntax, but beyond that the choice is pretty arbitrary. It's not important that the representation be the Right® choice. It's just important that it be a common, agreed-upon choice so that there can be a rich community of program-manipulating programs that do trade in this common representation. This is probably the best quote about homoiconicity I've read anywhere and explains why the term is tricky to pin down. On Fri, Jul 24, 2015 at 2:21 PM, Brandon Taylor brandon@gmail.com javascript: wrote: Macros seem to have a bunch of limitations. They can't be overloaded to take advantage of Julia's typing system. Their functionality can somewhat mimic evaluation within a function in its direct parent environment, but what if you want to write functions within your macro, or macros inside your macro, etc? Probably still possible to do many things, but it would invovle passing expressions back in forth in confusing ways, where as with environments, as long as an environment is kept attached to every expression, they can be immediately evaluated in any sub or subsub etc. environment.
Re: [julia-users] Re: Environment reification and lazy evaluation
Le vendredi 24 juillet 2015 à 08:48 -0700, Brandon Taylor a écrit :
Ok, maybe I was being harsh when I used the word mis-marketing. The
post here: http://julialang.org/blog/2012/02/why-we-created-julia/ sp
ecifically mentions that one of Julia's goals is to be as easy for
statistics as R. Building a domain specific language is more
difficult (but clearly not impossible) without environment access.
Again, the examples I'm most familiar with are in the Hadleyverse
(see previous posts in this thread for links to vignettes). Also,
this is probably the fifth time that me being stubborn seems to have
caused tensions on threads, and I'm sorry for that. Learning a bit
about C compilers convinced me that I was completely wrong about
environments needing to be implemented for scoping to work.
No worries, but that still doesn't give me a very concrete example of
what you're trying to do. :-)
I'm asking because if you only want to do things like dplyr offers,
you'll likely work only on data frames or similar structures. Then I
don't think you really need reified environments. DataFramesMeta.jl
already shows that macros can be quite powerful to reproduce dplyr-like
features the Julian way. If you show us a use case where this strategy
wouldn't work, maybe we can make more helpful comments.
Regards
On Friday, July 24, 2015 at 3:46:28 AM UTC-4, Milan Bouchet-Valat
wrote:
Le jeudi 23 juillet 2015 à 15:31 -0700, Brandon Taylor a écrit :
I'm not saying inherent slowness will be particularly useful. I'm
saying environment access will be particularly useful. The point
of
Julia is no compromise between speed and code-ability. If it is
not
possible to integrate environment access and speed, then I think
that
Julia is mis-marketing itself, at least as a viable R
alternative.
Could you give us a concrete example of what you would like to do
with
environments? I haven't encountered cases where I would personally
miss
this R feature in Julia, so I don't understand what you're trying
to
achieve.
Also, note that when we say that a solution will be slow in Julia,
it
doesn't mean that it would be slower than in R. It's just that to
write
code approximately equivalent to C, you cannot use the same
solutions
as R.
Regards
On Friday, July 24, 2015 at 12:35:13 AM UTC+4, Yichao Yu wrote:
On Thu, Jul 23, 2015 at 4:23 PM, Brandon Taylor
brandon@gmail.com wrote:
Will it cause a slowdown because there is a better way to do
it
or will it
cause a slowdown because something inherent about environment
access?
Because if the second is the case, then it would be worth it,
at
least to
me.
AFAICT, environment access is basically global variable and it
will
trash any optimization julia can do so in some sense the slow
down
is
inherent about environment access.
I don't see why the inherent slowness would be particularly
useful
though.
On Thursday, July 23, 2015 at 11:13:14 AM UTC+8, Yichao Yu
wrote:
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, made some progress. Still having trouble with a _ENV_
not
defined
error:
how could it not be defined if it's a global variable???
Figuring out
argument passing is going to be tricky. Is this kind of
system
feasible
or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the
parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global,
Re: [julia-users] Re: Environment reification and lazy evaluation
Macros seem to have a bunch of limitations. They can't be overloaded to take advantage of Julia's typing system. Their functionality can somewhat mimic evaluation within a function in its direct parent environment, but what if you want to write functions within your macro, or macros inside your macro, etc? Probably still possible to do many things, but it would invovle passing expressions back in forth in confusing ways, where as with environments, as long as an environment is kept attached to every expression, they can be immediately evaluated in any sub or subsub etc. environment.
Re: [julia-users] Re: Environment reification and lazy evaluation
Macros seem to have a bunch of limitations. They can't be overloaded to take advantage of Julia's typing system. Their functionality can somewhat mimic evaluation within a function in its direct parent environment, but what if you want to write functions within your macro, or macros inside your macro, etc? Probably still possible to do many things, but it would invovle passing expressions back in forth in confusing ways, where as with environments, as long as an environment is kept attached to every expression, they can be immediately evaluated in any sub or subsub etc. environment.
Re: [julia-users] Re: Environment reification and lazy evaluation
Lisp has already been down this path – fexprs https://en.wikipedia.org/wiki/Fexpr were special forms with behavior similar to R's functions, and they were deprecated in the 1980s in favor of macros, https://en.wikipedia.org/wiki/Fexpr#Mainstream_use_and_deprecation: His central objection was that, in a Lisp dialect that allows fexprs, static analysis cannot determine generally whether an operator represents an ordinary function or a fexpr — therefore, static analysis cannot determine whether or not the operands will be evaluated. In particular, the compiler cannot tell whether a subexpression can be safely optimized, since the subexpression might be treated as unevaluated data at run-time. If Lispers think a feature is too dynamic and hard to understand, that's a danger sign. In general, I wouldn't want to argue too much with Kent Pitman :-) On the subject of Kent Pitman, here's a random bit of Lisp wisdom from this interview http://developers.slashdot.org/story/01/11/03/1726251/kent-m-pitman-answers-on-lisp-and-much-more with him: I like Lisp's willingness to represent itself. People often explain this as its ability to represent itself, but I think that's wrong. Most languages are capable of representing themselves, but they simply don't have the will to. Lisp programs are represented by lists and programmers are aware of that. It wouldn't matter if it had been arrays. It does matter that it's program structure that is represented, and not character syntax, but beyond that the choice is pretty arbitrary. It's not important that the representation be the Right® choice. It's just important that it be a common, agreed-upon choice so that there can be a rich community of program-manipulating programs that do trade in this common representation. This is probably the best quote about homoiconicity I've read anywhere and explains why the term is tricky to pin down. On Fri, Jul 24, 2015 at 2:21 PM, Brandon Taylor brandon.taylor...@gmail.com wrote: Macros seem to have a bunch of limitations. They can't be overloaded to take advantage of Julia's typing system. Their functionality can somewhat mimic evaluation within a function in its direct parent environment, but what if you want to write functions within your macro, or macros inside your macro, etc? Probably still possible to do many things, but it would invovle passing expressions back in forth in confusing ways, where as with environments, as long as an environment is kept attached to every expression, they can be immediately evaluated in any sub or subsub etc. environment.
Re: [julia-users] Re: Environment reification and lazy evaluation
Le vendredi 24 juillet 2015 à 11:21 -0700, Brandon Taylor a écrit : Macros seem to have a bunch of limitations. They can't be overloaded to take advantage of Julia's typing system. Their functionality can somewhat mimic evaluation within a function in its direct parent environment, but what if you want to write functions within your macro, or macros inside your macro, etc? Probably still possible to do many things, but it would invovle passing expressions back in forth in confusing ways, where as with environments, as long as an environment is kept attached to every expression, they can be immediately evaluated in any sub or subsub etc. environment. I'm still looking for examples of situations where these limitations are indeed a problem... Discussing APIs in full generality, without concrete applications, is very hard. Regards
Re: [julia-users] Re: Environment reification and lazy evaluation
I agree, it would be very helpful to the discussion to have some examples of the problems Brandon is looking to solve. On Fri, Jul 24, 2015 at 3:09 PM, Milan Bouchet-Valat nalimi...@club.fr wrote: Le vendredi 24 juillet 2015 à 11:21 -0700, Brandon Taylor a écrit : Macros seem to have a bunch of limitations. They can't be overloaded to take advantage of Julia's typing system. Their functionality can somewhat mimic evaluation within a function in its direct parent environment, but what if you want to write functions within your macro, or macros inside your macro, etc? Probably still possible to do many things, but it would invovle passing expressions back in forth in confusing ways, where as with environments, as long as an environment is kept attached to every expression, they can be immediately evaluated in any sub or subsub etc. environment. I'm still looking for examples of situations where these limitations are indeed a problem... Discussing APIs in full generality, without concrete applications, is very hard. Regards
Re: [julia-users] Re: Environment reification and lazy evaluation
On Thu, Jul 23, 2015 at 4:23 PM, Brandon Taylor
brandon.taylor...@gmail.com wrote:
Will it cause a slowdown because there is a better way to do it or will it
cause a slowdown because something inherent about environment access?
Because if the second is the case, then it would be worth it, at least to
me.
AFAICT, environment access is basically global variable and it will
trash any optimization julia can do so in some sense the slow down is
inherent about environment access.
I don't see why the inherent slowness would be particularly useful though.
On Thursday, July 23, 2015 at 11:13:14 AM UTC+8, Yichao Yu wrote:
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, made some progress. Still having trouble with a _ENV_ not defined
error:
how could it not be defined if it's a global variable??? Figuring out
argument passing is going to be tricky. Is this kind of system feasible
or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict daughter of
the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict daughter of
the
old environment with dict contents
macro _ADD_ENV_MACRO_(dict)
esc(quote
_DICT_ = $dict
_ADD_ENV_ = gensym()
eval(quote
$_ADD_ENV_ = convert(Dict, $_DICT_)
$_ADD_ENV_[:_parent] = $(Expr(:quote, _ENV_))
end)
_ENV_ = _ADD_ENV_
end)
end
# jump back in time to the previous generation
macro _REMOVE_ENV_MACRO_()
esc(quote
_ENV_ = eval(_ENV_)[:_parent]
end)
end
# new types will have to be included in a module at the beginning of
code
# that module will need to be converted to a Dict along with base
# a namespace will need to be created such that module dicts inherit
from
each other, with base at the top
@_ENV_MACRO_()
@_ADD_ENV_MACRO_(Base)
@_NEW_ENV_MACRO_
# test expression
e =
quote
a = 1
b = 2
test = function()
b = a
end
end
# reformat code to use dict scoping
function _ENV_REPLACE_(_Lazy_::_LAZY_)
e = copy(_Lazy_._E_)
_ENV_ = _Lazy_._ENV_
# expressions wrapped in _esc will be left alone
if length(e.args) 0
if (e.head == :call) (e.args[1] == :_esc)
return e.args[2]
end
end
# set a new scope for a new function. This will also have to be done
with
for loops, modules, etc.
if (e.head == :function)
# insert a new scope definition into the function definition
e.args[2].args = [
e.args[2].args[1],
:(@_NEW_ENV_MACRO_),
:(_ENV_REPLACE!_(
$(Expr(:block,
e.args[2].args[2:end]...,
:(@_REMOVE_ENV_MACRO_)]
# ignore line numbers
elseif e.head != :line
# for each sentence
for i in 1:length(e.args)
# replace symbols with their dict scoped version
if typeof(e.args[i]) == Symbol
#avoid types
if !(e.args[i] in _TYPES_)
e.args[i] = :($_ENV_[$(string(e.args[i]))])
end
# recur into new expressions
elseif typeof(e.args[i]) == Expr
e.args[i] = _ENV_REPLACE!_(e.args[i], _ENV_)
end
end
end
e
end
function _LAZY_(e::Expr)
_LAZY_(e, _ENV_)
end
macro _LAZY_EVAL_(_Lazy_)
esc(quote
eval(_ENV_REPLACE_($_Lazy_))
Re: [julia-users] Re: Environment reification and lazy evaluation
Will it cause a slowdown because there is a better way to do it or will it
cause a slowdown because something inherent about environment access?
Because if the second is the case, then it would be worth it, at least to
me.
On Thursday, July 23, 2015 at 11:13:14 AM UTC+8, Yichao Yu wrote:
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon@gmail.com javascript: wrote:
Ok, made some progress. Still having trouble with a _ENV_ not defined
error:
how could it not be defined if it's a global variable??? Figuring out
argument passing is going to be tricky. Is this kind of system feasible
or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict daughter of
the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict daughter of
the
old environment with dict contents
macro _ADD_ENV_MACRO_(dict)
esc(quote
_DICT_ = $dict
_ADD_ENV_ = gensym()
eval(quote
$_ADD_ENV_ = convert(Dict, $_DICT_)
$_ADD_ENV_[:_parent] = $(Expr(:quote, _ENV_))
end)
_ENV_ = _ADD_ENV_
end)
end
# jump back in time to the previous generation
macro _REMOVE_ENV_MACRO_()
esc(quote
_ENV_ = eval(_ENV_)[:_parent]
end)
end
# new types will have to be included in a module at the beginning of
code
# that module will need to be converted to a Dict along with base
# a namespace will need to be created such that module dicts inherit
from
each other, with base at the top
@_ENV_MACRO_()
@_ADD_ENV_MACRO_(Base)
@_NEW_ENV_MACRO_
# test expression
e =
quote
a = 1
b = 2
test = function()
b = a
end
end
# reformat code to use dict scoping
function _ENV_REPLACE_(_Lazy_::_LAZY_)
e = copy(_Lazy_._E_)
_ENV_ = _Lazy_._ENV_
# expressions wrapped in _esc will be left alone
if length(e.args) 0
if (e.head == :call) (e.args[1] == :_esc)
return e.args[2]
end
end
# set a new scope for a new function. This will also have to be done
with
for loops, modules, etc.
if (e.head == :function)
# insert a new scope definition into the function definition
e.args[2].args = [
e.args[2].args[1],
:(@_NEW_ENV_MACRO_),
:(_ENV_REPLACE!_(
$(Expr(:block,
e.args[2].args[2:end]...,
:(@_REMOVE_ENV_MACRO_)]
# ignore line numbers
elseif e.head != :line
# for each sentence
for i in 1:length(e.args)
# replace symbols with their dict scoped version
if typeof(e.args[i]) == Symbol
#avoid types
if !(e.args[i] in _TYPES_)
e.args[i] = :($_ENV_[$(string(e.args[i]))])
end
# recur into new expressions
elseif typeof(e.args[i]) == Expr
e.args[i] = _ENV_REPLACE!_(e.args[i], _ENV_)
end
end
end
e
end
function _LAZY_(e::Expr)
_LAZY_(e, _ENV_)
end
macro _LAZY_EVAL_(_Lazy_)
esc(quote
eval(_ENV_REPLACE_($_Lazy_))
end)
end
@_LAZY_EVAL_(_LAZY_(e))
eval(_ENV_)[a]
eval(_ENV_)[b]
eval(_ENV_)[test]() ## ERROR HERE
On
Re: [julia-users] Re: Environment reification and lazy evaluation
What kind of mismarketing do you think is happening? I've been pretty clear on
my position (which seems to be representative of others here): environment
reification is a bad idea. It's basically just a tempting hack based on the
fact that slow language implementations often use dictionaries to implement
environments – if the environment is just a dictionary, why not give people
access to that dictionary? But that's not how environments are represented in
fast language implementations. Once you've taken the bait, though, you're stuck
with the slow implementation, or trying to emulate it somehow. Rather than
willfully putting ourselves in the slow lane, we're opting not to take the
performance-poisoning bait in the first place.
On Jul 23, 2015, at 6:31 PM, Brandon Taylor brandon.taylor...@gmail.com
wrote:
I'm not saying inherent slowness will be particularly useful. I'm saying
environment access will be particularly useful. The point of Julia is no
compromise between speed and code-ability. If it is not possible to integrate
environment access and speed, then I think that Julia is mis-marketing
itself, at least as a viable R alternative.
On Friday, July 24, 2015 at 12:35:13 AM UTC+4, Yichao Yu wrote:
On Thu, Jul 23, 2015 at 4:23 PM, Brandon Taylor
brandon@gmail.com wrote:
Will it cause a slowdown because there is a better way to do it or will it
cause a slowdown because something inherent about environment access?
Because if the second is the case, then it would be worth it, at least to
me.
AFAICT, environment access is basically global variable and it will
trash any optimization julia can do so in some sense the slow down is
inherent about environment access.
I don't see why the inherent slowness would be particularly useful though.
On Thursday, July 23, 2015 at 11:13:14 AM UTC+8, Yichao Yu wrote:
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, made some progress. Still having trouble with a _ENV_ not defined
error:
how could it not be defined if it's a global variable??? Figuring out
argument passing is going to be tricky. Is this kind of system feasible
or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict daughter of
the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict daughter of
the
old environment with dict contents
macro _ADD_ENV_MACRO_(dict)
esc(quote
_DICT_ = $dict
_ADD_ENV_ = gensym()
eval(quote
$_ADD_ENV_ = convert(Dict, $_DICT_)
$_ADD_ENV_[:_parent] = $(Expr(:quote, _ENV_))
end)
_ENV_ = _ADD_ENV_
end)
end
# jump back in time to the previous generation
macro _REMOVE_ENV_MACRO_()
esc(quote
_ENV_ = eval(_ENV_)[:_parent]
end)
end
# new types will have to be included in a module at the beginning of
code
# that module will need to be converted to a Dict along with base
# a namespace will need to be created such that module dicts inherit
from
each other, with base at the top
@_ENV_MACRO_()
@_ADD_ENV_MACRO_(Base)
@_NEW_ENV_MACRO_
Re: [julia-users] Re: Environment reification and lazy evaluation
I'm not saying inherent slowness will be particularly useful. I'm saying
environment access will be particularly useful. The point of Julia is no
compromise between speed and code-ability. If it is not possible to
integrate environment access and speed, then I think that Julia is
mis-marketing itself, at least as a viable R alternative.
On Friday, July 24, 2015 at 12:35:13 AM UTC+4, Yichao Yu wrote:
On Thu, Jul 23, 2015 at 4:23 PM, Brandon Taylor
brandon@gmail.com javascript: wrote:
Will it cause a slowdown because there is a better way to do it or will
it
cause a slowdown because something inherent about environment access?
Because if the second is the case, then it would be worth it, at least
to
me.
AFAICT, environment access is basically global variable and it will
trash any optimization julia can do so in some sense the slow down is
inherent about environment access.
I don't see why the inherent slowness would be particularly useful though.
On Thursday, July 23, 2015 at 11:13:14 AM UTC+8, Yichao Yu wrote:
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, made some progress. Still having trouble with a _ENV_ not defined
error:
how could it not be defined if it's a global variable??? Figuring out
argument passing is going to be tricky. Is this kind of system
feasible
or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point it to
an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict daughter of
the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict daughter of
the
old environment with dict contents
macro _ADD_ENV_MACRO_(dict)
esc(quote
_DICT_ = $dict
_ADD_ENV_ = gensym()
eval(quote
$_ADD_ENV_ = convert(Dict, $_DICT_)
$_ADD_ENV_[:_parent] = $(Expr(:quote, _ENV_))
end)
_ENV_ = _ADD_ENV_
end)
end
# jump back in time to the previous generation
macro _REMOVE_ENV_MACRO_()
esc(quote
_ENV_ = eval(_ENV_)[:_parent]
end)
end
# new types will have to be included in a module at the beginning of
code
# that module will need to be converted to a Dict along with base
# a namespace will need to be created such that module dicts inherit
from
each other, with base at the top
@_ENV_MACRO_()
@_ADD_ENV_MACRO_(Base)
@_NEW_ENV_MACRO_
# test expression
e =
quote
a = 1
b = 2
test = function()
b = a
end
end
# reformat code to use dict scoping
function _ENV_REPLACE_(_Lazy_::_LAZY_)
e = copy(_Lazy_._E_)
_ENV_ = _Lazy_._ENV_
# expressions wrapped in _esc will be left alone
if length(e.args) 0
if (e.head == :call) (e.args[1] == :_esc)
return e.args[2]
end
end
# set a new scope for a new function. This will also have to be
done
with
for loops, modules, etc.
if (e.head == :function)
# insert a new scope definition into the function definition
e.args[2].args = [
e.args[2].args[1],
:(@_NEW_ENV_MACRO_),
:(_ENV_REPLACE!_(
Re: [julia-users] Re: Environment reification and lazy evaluation
Ok, made some progress. Still having trouble with a _ENV_ not defined
error: how could it not be defined if it's a global variable??? Figuring
out argument passing is going to be tricky. Is this kind of system feasible
or is it going to cause huge slowdowns?
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict daughter of the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict daughter of the
old environment with dict contents
macro _ADD_ENV_MACRO_(dict)
esc(quote
_DICT_ = $dict
_ADD_ENV_ = gensym()
eval(quote
$_ADD_ENV_ = convert(Dict, $_DICT_)
$_ADD_ENV_[:_parent] = $(Expr(:quote, _ENV_))
end)
_ENV_ = _ADD_ENV_
end)
end
# jump back in time to the previous generation
macro _REMOVE_ENV_MACRO_()
esc(quote
_ENV_ = eval(_ENV_)[:_parent]
end)
end
# new types will have to be included in a module at the beginning of code
# that module will need to be converted to a Dict along with base
# a namespace will need to be created such that module dicts inherit from
each other, with base at the top
@_ENV_MACRO_()
@_ADD_ENV_MACRO_(Base)
@_NEW_ENV_MACRO_
# test expression
e =
quote
a = 1
b = 2
test = function()
b = a
end
end
# reformat code to use dict scoping
function _ENV_REPLACE_(_Lazy_::_LAZY_)
e = copy(_Lazy_._E_)
_ENV_ = _Lazy_._ENV_
# expressions wrapped in _esc will be left alone
if length(e.args) 0
if (e.head == :call) (e.args[1] == :_esc)
return e.args[2]
end
end
# set a new scope for a new function. This will also have to be done with
for loops, modules, etc.
if (e.head == :function)
# insert a new scope definition into the function definition
e.args[2].args = [
e.args[2].args[1],
:(@_NEW_ENV_MACRO_),
:(_ENV_REPLACE!_(
$(Expr(:block,
e.args[2].args[2:end]...,
:(@_REMOVE_ENV_MACRO_)]
# ignore line numbers
elseif e.head != :line
# for each sentence
for i in 1:length(e.args)
# replace symbols with their dict scoped version
if typeof(e.args[i]) == Symbol
#avoid types
if !(e.args[i] in _TYPES_)
e.args[i] = :($_ENV_[$(string(e.args[i]))])
end
# recur into new expressions
elseif typeof(e.args[i]) == Expr
e.args[i] = _ENV_REPLACE!_(e.args[i], _ENV_)
end
end
end
e
end
function _LAZY_(e::Expr)
_LAZY_(e, _ENV_)
end
macro _LAZY_EVAL_(_Lazy_)
esc(quote
eval(_ENV_REPLACE_($_Lazy_))
end)
end
@_LAZY_EVAL_(_LAZY_(e))
eval(_ENV_)[a]
eval(_ENV_)[b]
eval(_ENV_)[test]() ## ERROR HERE
On Wednesday, July 22, 2015 at 11:30:10 AM UTC+8, Brandon Taylor wrote:
More to do:
Expressions would also have to be escaped from quoting.
If we can't scope types within dicts, it might be necessary to have
special markers for types so they can avoid being scoped.
I don't think that macros will be necessary anymore
On Wednesday, July 22, 2015 at 11:14:38 AM UTC+8, Brandon Taylor wrote:
Ok so I've got a good start. I bet John Myles White didn't think I could
get this far. Anyway, I'm getting caught up in defining my own escape
function. I'm getting lost in multiple layers of meta.
using DataFrames
import Base.convert
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
end
dict
end
base_dict = convert(Dict, Base)
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in
Re: [julia-users] Re: Environment reification and lazy evaluation
On Wed, Jul 22, 2015 at 10:51 PM, Brandon Taylor
brandon.taylor...@gmail.com wrote:
Ok, made some progress. Still having trouble with a _ENV_ not defined error:
how could it not be defined if it's a global variable??? Figuring out
argument passing is going to be tricky. Is this kind of system feasible or
is it going to cause huge slowdowns?
It'll cause a huge slowdown.
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) : DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict daughter of the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent = $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict daughter of the
old environment with dict contents
macro _ADD_ENV_MACRO_(dict)
esc(quote
_DICT_ = $dict
_ADD_ENV_ = gensym()
eval(quote
$_ADD_ENV_ = convert(Dict, $_DICT_)
$_ADD_ENV_[:_parent] = $(Expr(:quote, _ENV_))
end)
_ENV_ = _ADD_ENV_
end)
end
# jump back in time to the previous generation
macro _REMOVE_ENV_MACRO_()
esc(quote
_ENV_ = eval(_ENV_)[:_parent]
end)
end
# new types will have to be included in a module at the beginning of code
# that module will need to be converted to a Dict along with base
# a namespace will need to be created such that module dicts inherit from
each other, with base at the top
@_ENV_MACRO_()
@_ADD_ENV_MACRO_(Base)
@_NEW_ENV_MACRO_
# test expression
e =
quote
a = 1
b = 2
test = function()
b = a
end
end
# reformat code to use dict scoping
function _ENV_REPLACE_(_Lazy_::_LAZY_)
e = copy(_Lazy_._E_)
_ENV_ = _Lazy_._ENV_
# expressions wrapped in _esc will be left alone
if length(e.args) 0
if (e.head == :call) (e.args[1] == :_esc)
return e.args[2]
end
end
# set a new scope for a new function. This will also have to be done with
for loops, modules, etc.
if (e.head == :function)
# insert a new scope definition into the function definition
e.args[2].args = [
e.args[2].args[1],
:(@_NEW_ENV_MACRO_),
:(_ENV_REPLACE!_(
$(Expr(:block,
e.args[2].args[2:end]...,
:(@_REMOVE_ENV_MACRO_)]
# ignore line numbers
elseif e.head != :line
# for each sentence
for i in 1:length(e.args)
# replace symbols with their dict scoped version
if typeof(e.args[i]) == Symbol
#avoid types
if !(e.args[i] in _TYPES_)
e.args[i] = :($_ENV_[$(string(e.args[i]))])
end
# recur into new expressions
elseif typeof(e.args[i]) == Expr
e.args[i] = _ENV_REPLACE!_(e.args[i], _ENV_)
end
end
end
e
end
function _LAZY_(e::Expr)
_LAZY_(e, _ENV_)
end
macro _LAZY_EVAL_(_Lazy_)
esc(quote
eval(_ENV_REPLACE_($_Lazy_))
end)
end
@_LAZY_EVAL_(_LAZY_(e))
eval(_ENV_)[a]
eval(_ENV_)[b]
eval(_ENV_)[test]() ## ERROR HERE
On Wednesday, July 22, 2015 at 11:30:10 AM UTC+8, Brandon Taylor wrote:
More to do:
Expressions would also have to be escaped from quoting.
If we can't scope types within dicts, it might be necessary to have
special markers for types so they can avoid being scoped.
I don't think that macros will be necessary anymore
On Wednesday, July 22, 2015 at 11:14:38 AM UTC+8, Brandon Taylor wrote:
Ok so I've got a good start. I bet John Myles White didn't think I could
get this far. Anyway, I'm getting caught up in defining my own escape
function. I'm getting lost in multiple layers of meta.
using DataFrames
import Base.convert
# allow inheritance from modules
function convert(::Type{Dict},
Re: [julia-users] Re: Environment reification and lazy evaluation
More to do:
Expressions would also have to be escaped from quoting.
If we can't scope types within dicts, it might be necessary to have special
markers for types so they can avoid being scoped.
I don't think that macros will be necessary anymore
On Wednesday, July 22, 2015 at 11:14:38 AM UTC+8, Brandon Taylor wrote:
Ok so I've got a good start. I bet John Myles White didn't think I could
get this far. Anyway, I'm getting caught up in defining my own escape
function. I'm getting lost in multiple layers of meta.
using DataFrames
import Base.convert
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
end
dict
end
base_dict = convert(Dict, Base)
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
end
dict
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(h[:_parent], key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# test expression
e =
quote
a = 1
# anonymous functions required for proper scoping
test = function()
b = a
end
end
# set up the global environment
_env = gensym()
eval(:($_env = [:_parent = base_dict] ) )
#_new_env = _env
# this is the code that needs to be escaped
#eval(:($_new_env = [:_parent = eval(_env)] ) )
# reformat code to use dict scoping
function env_replace!(e::Expr,
_env::Symbol = _env)
# expressions wrapped in _esc will be left alone
if length(e.args) 0
if (e.head == :call) (e.args[1] == :_esc)
return e.args[2]
end
end
# set a new scope for a new function. This will also have to be done
with for loops, modules, etc.
if (e.head == :function)
# insert a new scope definition into the function definition
_new_env = gensym()
e.args[2].args = [
e.args[2].args[1],
:_esc(), need help here ###
e.args[2].args[2:end] ]
_env = _new_env
end
# ignore line numbers
if e.head != :line
for i in 1:length(e.args)
# replace symbols with their dict scoped version
if typeof(e.args[i]) == Symbol
e.args[i] = :($_env[$(string(e.args[i]))])
# recur into new expressions
elseif typeof(e.args[i]) == Expr
e.args[i] = env_replace!(e.args[i], _env)
end
end
end
e
end
# here is an eval that allows evaluation within a certain dict scope
function lazy_eval(e::Expr,
_env::Symbol = _env)
eval(env_replace!(e), _env)
end
## TO DO
# fix _esc problem
# prevent environment symbols from being scoped (perhaps with a special
marker)
# rescope for, while, try, catch, finally, let, and type
# perhaps use fast anonymous to avoid performance slowdowns?
On Tuesday, July 21, 2015 at 10:10:58 AM UTC+8, Brandon Taylor wrote:
And there would need to be a special marker for them, such that if I'm in
function f, f[:a] won't get preprocessed as f[:f][:a]
On Tuesday, July 21, 2015 at 10:03:08 AM UTC+8, Brandon Taylor wrote:
Although that would probably require nested dicts. Each would have a
parent dict, and if a lookup isn't found in the current dict, the parent
dict would be searched.
On Tuesday, July 21, 2015 at 9:53:50 AM UTC+8, Brandon Taylor wrote:
I should be possible to preprocess code such that everything is put
into a dict based on the name of enclosing function (and global variables
will just go into a dict called global).
On Tuesday, July 21, 2015 at 9:42:00 AM UTC+8, Brandon Taylor wrote:
Dicts seem to work pretty well for this kind of thing.
On Tuesday, July 21, 2015 at 9:38:36 AM UTC+8, Brandon Taylor wrote:
I'm getting a cannot assign variables in other modules error.
On Tuesday, July 21, 2015 at 6:39:44 AM UTC+8, Yichao Yu wrote:
On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, a thought, Julia has an inbuilt idea of a module. Would it be
possible
to hijack this functionality to provide pseudo-environments? That
is, never
referring to anything that is not already in an explicit module?
And also,
have a data-frame simply be a module?
I think this would in principle works. A module is basically what
global scope means so all the performance concern applies.
On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor
wrote:
I don't know if you came across the vignette?
http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html
?
dplyr uses lazyeval extensively, see
Re: [julia-users] Re: Environment reification and lazy evaluation
Ok so I've got a good start. I bet John Myles White didn't think I could
get this far. Anyway, I'm getting caught up in defining my own escape
function. I'm getting lost in multiple layers of meta.
using DataFrames
import Base.convert
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
end
dict
end
base_dict = convert(Dict, Base)
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
end
dict
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index 0
if :_parent in keys(h)
Base.getindex(h[:_parent], key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# test expression
e =
quote
a = 1
# anonymous functions required for proper scoping
test = function()
b = a
end
end
# set up the global environment
_env = gensym()
eval(:($_env = [:_parent = base_dict] ) )
#_new_env = _env
# this is the code that needs to be escaped
#eval(:($_new_env = [:_parent = eval(_env)] ) )
# reformat code to use dict scoping
function env_replace!(e::Expr,
_env::Symbol = _env)
# expressions wrapped in _esc will be left alone
if length(e.args) 0
if (e.head == :call) (e.args[1] == :_esc)
return e.args[2]
end
end
# set a new scope for a new function. This will also have to be done with
for loops, modules, etc.
if (e.head == :function)
# insert a new scope definition into the function definition
_new_env = gensym()
e.args[2].args = [
e.args[2].args[1],
:_esc(), need help here ###
e.args[2].args[2:end] ]
_env = _new_env
end
# ignore line numbers
if e.head != :line
for i in 1:length(e.args)
# replace symbols with their dict scoped version
if typeof(e.args[i]) == Symbol
e.args[i] = :($_env[$(string(e.args[i]))])
# recur into new expressions
elseif typeof(e.args[i]) == Expr
e.args[i] = env_replace!(e.args[i], _env)
end
end
end
e
end
# here is an eval that allows evaluation within a certain dict scope
function lazy_eval(e::Expr,
_env::Symbol = _env)
eval(env_replace!(e), _env)
end
## TO DO
# fix _esc problem
# prevent environment symbols from being scoped (perhaps with a special
marker)
# rescope for, while, try, catch, finally, let, and type
# perhaps use fast anonymous to avoid performance slowdowns?
On Tuesday, July 21, 2015 at 10:10:58 AM UTC+8, Brandon Taylor wrote:
And there would need to be a special marker for them, such that if I'm in
function f, f[:a] won't get preprocessed as f[:f][:a]
On Tuesday, July 21, 2015 at 10:03:08 AM UTC+8, Brandon Taylor wrote:
Although that would probably require nested dicts. Each would have a
parent dict, and if a lookup isn't found in the current dict, the parent
dict would be searched.
On Tuesday, July 21, 2015 at 9:53:50 AM UTC+8, Brandon Taylor wrote:
I should be possible to preprocess code such that everything is put into
a dict based on the name of enclosing function (and global variables will
just go into a dict called global).
On Tuesday, July 21, 2015 at 9:42:00 AM UTC+8, Brandon Taylor wrote:
Dicts seem to work pretty well for this kind of thing.
On Tuesday, July 21, 2015 at 9:38:36 AM UTC+8, Brandon Taylor wrote:
I'm getting a cannot assign variables in other modules error.
On Tuesday, July 21, 2015 at 6:39:44 AM UTC+8, Yichao Yu wrote:
On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor
brandon@gmail.com wrote:
Ok, a thought, Julia has an inbuilt idea of a module. Would it be
possible
to hijack this functionality to provide pseudo-environments? That
is, never
referring to anything that is not already in an explicit module?
And also,
have a data-frame simply be a module?
I think this would in principle works. A module is basically what
global scope means so all the performance concern applies.
On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor
wrote:
I don't know if you came across the vignette?
http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html
?
dplyr uses lazyeval extensively, see
http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html .
The cool
thing about being able to incorporate this kind of thing in Julia
would be
being able to use the self-reflection capabilities.
On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean
wrote:
On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor
wrote:
To walk back in time, you could say something like: compile this
like
this was is in line 8.
Re: [julia-users] Re: Environment reification and lazy evaluation
I'm getting a cannot assign variables in other modules error. On Tuesday, July 21, 2015 at 6:39:44 AM UTC+8, Yichao Yu wrote: On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor brandon@gmail.com javascript: wrote: Ok, a thought, Julia has an inbuilt idea of a module. Would it be possible to hijack this functionality to provide pseudo-environments? That is, never referring to anything that is not already in an explicit module? And also, have a data-frame simply be a module? I think this would in principle works. A module is basically what global scope means so all the performance concern applies. On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor wrote: I don't know if you came across the vignette? http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html ? dplyr uses lazyeval extensively, see http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html . The cool thing about being able to incorporate this kind of thing in Julia would be being able to use the self-reflection capabilities. On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line (https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime
Re: [julia-users] Re: Environment reification and lazy evaluation
On Mon, Jul 20, 2015 at 9:41 PM, Brandon Taylor brandon.taylor...@gmail.com wrote: Dicts seem to work pretty well for this kind of thing. For storage, maybe. On Tuesday, July 21, 2015 at 9:38:36 AM UTC+8, Brandon Taylor wrote: I'm getting a cannot assign variables in other modules error. You cannot assign to module. You need `eval` On Tuesday, July 21, 2015 at 6:39:44 AM UTC+8, Yichao Yu wrote: On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor brandon@gmail.com wrote: Ok, a thought, Julia has an inbuilt idea of a module. Would it be possible to hijack this functionality to provide pseudo-environments? That is, never referring to anything that is not already in an explicit module? And also, have a data-frame simply be a module? I think this would in principle works. A module is basically what global scope means so all the performance concern applies. On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor wrote: I don't know if you came across the vignette? http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html ? dplyr uses lazyeval extensively, see http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html . The cool thing about being able to incorporate this kind of thing in Julia would be being able to use the self-reflection capabilities. On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line (https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are
Re: [julia-users] Re: Environment reification and lazy evaluation
I should be possible to preprocess code such that everything is put into a dict based on the name of enclosing function (and global variables will just go into a dict called global). On Tuesday, July 21, 2015 at 9:42:00 AM UTC+8, Brandon Taylor wrote: Dicts seem to work pretty well for this kind of thing. On Tuesday, July 21, 2015 at 9:38:36 AM UTC+8, Brandon Taylor wrote: I'm getting a cannot assign variables in other modules error. On Tuesday, July 21, 2015 at 6:39:44 AM UTC+8, Yichao Yu wrote: On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor brandon@gmail.com wrote: Ok, a thought, Julia has an inbuilt idea of a module. Would it be possible to hijack this functionality to provide pseudo-environments? That is, never referring to anything that is not already in an explicit module? And also, have a data-frame simply be a module? I think this would in principle works. A module is basically what global scope means so all the performance concern applies. On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor wrote: I don't know if you came across the vignette? http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html ? dplyr uses lazyeval extensively, see http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html . The cool thing about being able to incorporate this kind of thing in Julia would be being able to use the self-reflection capabilities. On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line (https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler
Re: [julia-users] Re: Environment reification and lazy evaluation
Although that would probably require nested dicts. Each would have a parent dict, and if a lookup isn't found in the current dict, the parent dict would be searched. On Tuesday, July 21, 2015 at 9:53:50 AM UTC+8, Brandon Taylor wrote: I should be possible to preprocess code such that everything is put into a dict based on the name of enclosing function (and global variables will just go into a dict called global). On Tuesday, July 21, 2015 at 9:42:00 AM UTC+8, Brandon Taylor wrote: Dicts seem to work pretty well for this kind of thing. On Tuesday, July 21, 2015 at 9:38:36 AM UTC+8, Brandon Taylor wrote: I'm getting a cannot assign variables in other modules error. On Tuesday, July 21, 2015 at 6:39:44 AM UTC+8, Yichao Yu wrote: On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor brandon@gmail.com wrote: Ok, a thought, Julia has an inbuilt idea of a module. Would it be possible to hijack this functionality to provide pseudo-environments? That is, never referring to anything that is not already in an explicit module? And also, have a data-frame simply be a module? I think this would in principle works. A module is basically what global scope means so all the performance concern applies. On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor wrote: I don't know if you came across the vignette? http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html ? dplyr uses lazyeval extensively, see http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html . The cool thing about being able to incorporate this kind of thing in Julia would be being able to use the self-reflection capabilities. On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line (https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access
Re: [julia-users] Re: Environment reification and lazy evaluation
And there would need to be a special marker for them, such that if I'm in function f, f[:a] won't get preprocessed as f[:f][:a] On Tuesday, July 21, 2015 at 10:03:08 AM UTC+8, Brandon Taylor wrote: Although that would probably require nested dicts. Each would have a parent dict, and if a lookup isn't found in the current dict, the parent dict would be searched. On Tuesday, July 21, 2015 at 9:53:50 AM UTC+8, Brandon Taylor wrote: I should be possible to preprocess code such that everything is put into a dict based on the name of enclosing function (and global variables will just go into a dict called global). On Tuesday, July 21, 2015 at 9:42:00 AM UTC+8, Brandon Taylor wrote: Dicts seem to work pretty well for this kind of thing. On Tuesday, July 21, 2015 at 9:38:36 AM UTC+8, Brandon Taylor wrote: I'm getting a cannot assign variables in other modules error. On Tuesday, July 21, 2015 at 6:39:44 AM UTC+8, Yichao Yu wrote: On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor brandon@gmail.com wrote: Ok, a thought, Julia has an inbuilt idea of a module. Would it be possible to hijack this functionality to provide pseudo-environments? That is, never referring to anything that is not already in an explicit module? And also, have a data-frame simply be a module? I think this would in principle works. A module is basically what global scope means so all the performance concern applies. On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor wrote: I don't know if you came across the vignette? http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html ? dplyr uses lazyeval extensively, see http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html . The cool thing about being able to incorporate this kind of thing in Julia would be being able to use the self-reflection capabilities. On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line (https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon
Re: [julia-users] Re: Environment reification and lazy evaluation
Ok, a thought, Julia has an inbuilt idea of a module. Would it be possible to hijack this functionality to provide pseudo-environments? That is, never referring to anything that is not already in an explicit module? And also, have a data-frame simply be a module? On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor wrote: I don't know if you came across the vignette? http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html ? dplyr uses lazyeval extensively, see http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html . The cool thing about being able to incorporate this kind of thing in Julia would be being able to use the self-reflection capabilities. On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line ( https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4,
Re: [julia-users] Re: Environment reification and lazy evaluation
On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor brandon.taylor...@gmail.com wrote: Ok, a thought, Julia has an inbuilt idea of a module. Would it be possible to hijack this functionality to provide pseudo-environments? That is, never referring to anything that is not already in an explicit module? And also, have a data-frame simply be a module? I think this would in principle works. A module is basically what global scope means so all the performance concern applies. On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor wrote: I don't know if you came across the vignette? http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html ? dplyr uses lazyeval extensively, see http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html . The cool thing about being able to incorporate this kind of thing in Julia would be being able to use the self-reflection capabilities. On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line (https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible
Re: [julia-users] Re: Environment reification and lazy evaluation
Hi Cedric, a macroexpand shortcut for the REPL shouldn't be a big problem, I just tried to prototype one and it seems to work fine: https://github.com/meggart/julia/commit/73c96897572cedf10f74fd09907232c706bbdf48 Is there any special functionality/problems which I don't see that should be considered?
Re: [julia-users] Re: Environment reification and lazy evaluation
On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line ( https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4,
Re: [julia-users] Re: Environment reification and lazy evaluation
I don't know if you came across the vignette? http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html ? dplyr uses lazyeval extensively, see http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html . The cool thing about being able to incorporate this kind of thing in Julia would be being able to use the self-reflection capabilities. On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Could you provide context or a real-world use? I've looked at the lazyeval package, and I'm not entirely sure what it does. Does it provide lazy evaluation for R? That's easy to achieve in Julia (well, sorta). Instead of d = determinant(matrix) u = 2 * d you can write d = ()-determinant(matrix) u = 2 * d() # determinant is evaluated on use, in the context where it was originally defined With macros this can turn into d = lazy(determinant(matrix)) which looks nicer (and also can avoid computing the determinant twice if d() is called twice). Cédric On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line ( https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply
Re: [julia-users] Re: Environment reification and lazy evaluation
Hmm, so the reason that macros are able to function is that they occur before type computation? One solution might be to make scope annotations that override defaults. Then, when eval is called, it optionally attaches scope annotations to a given scope. Note that scope annotations should be able to refer not only to regular scopes but also indexed items like dicts and dataframes. Wait, how does eval handle type computation if it occurs after type computation? On Friday, July 10, 2015 at 5:51:05 AM UTC-4, Fabian Gans wrote: Hi Cedric, a macroexpand shortcut for the REPL shouldn't be a big problem, I just tried to prototype one and it seems to work fine: https://github.com/meggart/julia/commit/73c96897572cedf10f74fd09907232c706bbdf48 Is there any special functionality/problems which I don't see that should be considered?
Re: [julia-users] Re: Environment reification and lazy evaluation
Hmm, so the reason that macros are able to function is that they occur before type computation? I don't know how best to respond to this question, but I will throw out that staged functions are expanded after type information is known to the compiler. On Friday, July 10, 2015 at 10:07:23 AM UTC-4, Brandon Taylor wrote: Hmm, so the reason that macros are able to function is that they occur before type computation? One solution might be to make scope annotations that override defaults. Then, when eval is called, it optionally attaches scope annotations to a given scope. Note that scope annotations should be able to refer not only to regular scopes but also indexed items like dicts and dataframes. Wait, how does eval handle type computation if it occurs after type computation? On Friday, July 10, 2015 at 5:51:05 AM UTC-4, Fabian Gans wrote: Hi Cedric, a macroexpand shortcut for the REPL shouldn't be a big problem, I just tried to prototype one and it seems to work fine: https://github.com/meggart/julia/commit/73c96897572cedf10f74fd09907232c706bbdf48 Is there any special functionality/problems which I don't see that should be considered?
Re: [julia-users] Re: Environment reification and lazy evaluation
Right now the best resources to learn how the compiler works are the developer docs and Jeff's talk at JuliaCon last year. http://docs.julialang.org/en/latest/#developer-documentation https://www.youtube.com/watch?v=osdeT-tWjzk (there were several more compiler-related talks this year; hopefully the recordings will be posted soon) On Fri, Jul 10, 2015 at 10:07 AM, Brandon Taylor brandon.taylor...@gmail.com wrote: Hmm, so the reason that macros are able to function is that they occur before type computation? One solution might be to make scope annotations that override defaults. Then, when eval is called, it optionally attaches scope annotations to a given scope. Note that scope annotations should be able to refer not only to regular scopes but also indexed items like dicts and dataframes. Wait, how does eval handle type computation if it occurs after type computation? On Friday, July 10, 2015 at 5:51:05 AM UTC-4, Fabian Gans wrote: Hi Cedric, a macroexpand shortcut for the REPL shouldn't be a big problem, I just tried to prototype one and it seems to work fine: https://github.com/meggart/julia/commit/73c96897572cedf10f74fd09907232c706bbdf48 Is there any special functionality/problems which I don't see that should be considered?
Re: [julia-users] Re: Environment reification and lazy evaluation
On Friday, July 10, 2015 at 5:51:05 AM UTC-4, Fabian Gans wrote: Hi Cedric, a macroexpand shortcut for the REPL shouldn't be a big problem, I just tried to prototype one and it seems to work fine: https://github.com/meggart/julia/commit/73c96897572cedf10f74fd09907232c706bbdf48 Is there any special functionality/problems which I don't see that should be considered? Cool, thanks! I don't build from source, but I look forward to that support in the future. You can see how SLIME (the Common Lisp emacs support library) does it here https://www.youtube.com/watch?v=sBcPNr1CKKwlist=PL2HCA9vU95bsKlhegE6-_Wiks3EpzXuoFindex=4t=593. I use mostly IJulia these days, and I don't see a clean mechanism for integrating it in there, but Jupyter is moving at a steady pace, so maybe in a few releases. The main use for Emacs' macroexpand is to understand how a complex macro works in a given piece of code, and to debug new macros. The ability to selectively macroexpand and contract specific portions of the tree is important. Best, Cédric
Re: [julia-users] Re: Environment reification and lazy evaluation
On Thursday, July 9, 2015 at 10:25:24 AM UTC-4, Yichao Yu wrote: Julia already have `macroexpand` so I guess you are talking about editor/REPL feature? Yeah. Emacs' macroexpand makes a big difference in usability for macros IMO, otherwise one has to repeatedly cut and paste code in the REPL, which is no fun at all. On Wednesday, July 8, 2015 at 9:51:16 PM UTC-4, Brandon Taylor wrote: Edit: Hmm, so that means that any implementation of environments would have to be handled in two separate ways: one by the compiler for non-global scope, and one in run time for the special global system?] On Wednesday, July 8, 2015 at 9:49:50 PM UTC-4, Brandon Taylor wrote: Hmm, so that means that any implementation of environments would have to be handled in to separate ways: one by the compiler for non-global scope, and one in run time using a second for the special global system?] On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly
Re: [julia-users] Re: Environment reification and lazy evaluation
Environments are very natural in interpreted languages, but I'm not aware of any fully-compiled language that supports them because they'd be a major headache. I agree that macro behavior can be hard to predict. In Common Lisp with Emacs, one can macroexpand any expression anywhere with one keypress, and it is incredibly useful for figuring things out. I'm sure that Julia will eventually get that functionality, it's several orders of magnitude easier than supporting environments. On Wednesday, July 8, 2015 at 9:51:16 PM UTC-4, Brandon Taylor wrote: Edit: Hmm, so that means that any implementation of environments would have to be handled in two separate ways: one by the compiler for non-global scope, and one in run time for the special global system?] On Wednesday, July 8, 2015 at 9:49:50 PM UTC-4, Brandon Taylor wrote: Hmm, so that means that any implementation of environments would have to be handled in to separate ways: one by the compiler for non-global scope, and one in run time using a second for the special global system?] On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for
Re: [julia-users] Re: Environment reification and lazy evaluation
On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line ( https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different.
Re: [julia-users] Re: Environment reification and lazy evaluation
gc optimizations aren't really the critical issue (although it is a possible side-benefit). the big advantages are enforced type computability (they cannot be accessed in ways that are invisible to inference) and semantic purity (the variables are exactly those you see in the program, plus any spliced in from a macro). it's not quite right to consider this a compiler phase either, since the identification of a variable's scope is actually done at parse time (just after splicing in the return values from any macro). i agree that macros are confusing when encountering them for the first time. it sometimes helps to view them as functions that take code syntax as arguments and return modified code syntax. that return value is then spliced into the original code exactly as if it was the expression that you typed originally. it is also sometimes also helpful to note that this code transformation happens very early, just after parsing the code, and long before any variables exist or any code has been executed. On Friday, July 10, 2015 at 4:38:55 AM UTC+2, Brandon Taylor wrote: Is the issue garbage collection? That because by choosing variables dynamically from a symbol table, you don't know which variables are going to be used, so you don't know which data can be deleted early? On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Or maybe not, macros still confuse me. On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line ( https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist
Re: [julia-users] Re: Environment reification and lazy evaluation
To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Or maybe not, macros still confuse me. On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line ( https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was
Re: [julia-users] Re: Environment reification and lazy evaluation
Is the issue garbage collection? That because by choosing variables dynamically from a symbol table, you don't know which variables are going to be used, so you don't know which data can be deleted early? On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Or maybe not, macros still confuse me. On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line ( https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM
Re: [julia-users] Re: Environment reification and lazy evaluation
Instead of speculating about how Julia's compiler handles scope, perhaps it would be more fruitful to read the source code and learn how things currently work? Better still: why not spend some time trying to implement your feature request? If you can manage to implement scope reification without any performance degradations for any existing code, I think your work will be extremely well received In general, a working prototype is a sine qua non for this kind of sweeping redesign to be given serious consideration. -- John On Friday, July 10, 2015 at 4:38:55 AM UTC+2, Brandon Taylor wrote: Is the issue garbage collection? That because by choosing variables dynamically from a symbol table, you don't know which variables are going to be used, so you don't know which data can be deleted early? On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor wrote: To walk back in time, you could say something like: compile this like this was is in line 8. Or compile this like this was in line 5. It seems like Julia already has some of this functionality in macros. Internal variables are compiled as if they were in local scope. But escaped expressions are compiled as if they were in global scope. Or maybe not, macros still confuse me. On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean wrote: On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, Yes, more or less. This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. Could you expand on what you're thinking of? This kind of compile-time environment could conceivably be exposed to macros. Common Lisp had proposals along that line ( https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html) but as far as I can tell, it was too complicated and not useful enough, so it was axed/neutered at some point in the standardization process. Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment I don't know about R, but to me that sounds entirely doable with closures (and macros will give you a nice syntax for it) On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to
Re: [julia-users] Re: Environment reification and lazy evaluation
Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com javascript: wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com javascript: wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope
Re: [julia-users] Re: Environment reification and lazy evaluation
https://en.wikipedia.org/wiki/Variable_(computer_science)#Scope_and_extent On Thu, Jul 9, 2015 at 4:14 PM, Brandon Taylor brandon.taylor...@gmail.com wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the
Re: [julia-users] Re: Environment reification and lazy evaluation
Unless it did, in which case, I'm just uneducated. On Thursday, July 9, 2015 at 6:31:03 PM UTC-4, Brandon Taylor wrote: Ok, that was an interesting article, but it didn't really answer my question. On Thursday, July 9, 2015 at 4:20:45 PM UTC-4, Isaiah wrote: https://en.wikipedia.org/wiki/Variable_(computer_science)#Scope_and_extent On Thu, Jul 9, 2015 at 4:14 PM, Brandon Taylor brandon@gmail.com wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with
Re: [julia-users] Re: Environment reification and lazy evaluation
Ok, that was an interesting article, but it didn't really answer my question. On Thursday, July 9, 2015 at 4:20:45 PM UTC-4, Isaiah wrote: https://en.wikipedia.org/wiki/Variable_(computer_science)#Scope_and_extent On Thu, Jul 9, 2015 at 4:14 PM, Brandon Taylor brandon@gmail.com javascript: wrote: Ok, here's where I'm getting hung up. You said that the compiler figures out the creation/lifetime of all variables at compile time. So does that mean there's a list like: a maps to location 0 and exists from line 3 to line 9 b maps to location 1 and exists from line 7 to line 9 a maps to location 10 and exists from line 7 to 9? and that to map variables to locations on any particular line, the compiler works its way up the list, This is perhaps even more helpful than the environment. The environment is immediately and completely determinable at any point in the program. This could make it possible to walk back in time even within the same scope. On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so
Re: [julia-users] Re: Environment reification and lazy evaluation
Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
[julia-users] Re: Environment reification and lazy evaluation
For example, here's a code snippet from R: function (x, data) eval(x$expr, data, x$env) x is an expression which contains an environment attribute. It is evaluated first checking the bindings in data (data being a dataframe) and then checing the bindings in x's environment. On Wednesday, July 8, 2015 at 3:37:04 PM UTC-4, Brandon Taylor wrote: *reifying. Deifying environments might not be the best idea. On Wednesday, July 8, 2015 at 3:34:53 PM UTC-4, Brandon Taylor wrote: I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FJuliaStats%2FDataFrames.jl%2Fissues%2F504sa=Dsntz=1usg=AFQjCNHgUEZP8TyJ_BuUyyFA5SIxneOJTA I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
Re: [julia-users] Re: Environment reification and lazy evaluation
If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
[julia-users] Re: Environment reification and lazy evaluation
I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FJuliaStats%2FDataFrames.jl%2Fissues%2F504sa=Dsntz=1usg=AFQjCNHgUEZP8TyJ_BuUyyFA5SIxneOJTA I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
[julia-users] Re: Environment reification and lazy evaluation
It should be noted that in R, environments are simply pointers. They take up no memory and shouldn't cause drastic slowdowns (I don't think). On Wednesday, July 8, 2015 at 3:48:08 PM UTC-4, Brandon Taylor wrote: For example, here's a code snippet from R: function (x, data) eval(x$expr, data, x$env) x is an expression which contains an environment attribute. It is evaluated first checking the bindings in data (data being a dataframe) and then checing the bindings in x's environment. On Wednesday, July 8, 2015 at 3:37:04 PM UTC-4, Brandon Taylor wrote: *reifying. Deifying environments might not be the best idea. On Wednesday, July 8, 2015 at 3:34:53 PM UTC-4, Brandon Taylor wrote: I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FJuliaStats%2FDataFrames.jl%2Fissues%2F504sa=Dsntz=1usg=AFQjCNHgUEZP8TyJ_BuUyyFA5SIxneOJTA I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
Re: [julia-users] Re: Environment reification and lazy evaluation
Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
[julia-users] Re: Environment reification and lazy evaluation
In fact, if environments could be reifyed, I'd go so far as to say that
Julia should have a separate syntax for a) evaluating an expression in its
defining environment and b) evaluating an expression in its evaluation
environment.
On Wednesday, July 8, 2015 at 3:53:29 PM UTC-4, Brandon Taylor wrote:
For comparison, here is the analogous code from DataFramesMeta.
replace_syms(x, membernames) = x
function replace_syms(e::Expr, membernames)
if e.head == :call length(e.args) == 2 e.args[1] == :^
return e.args[2]
elseif e.head == :. # special case for :a.b
return Expr(e.head, replace_syms(e.args[1], membernames),
typeof(e.args[2]) == Expr e.args[2].head ==
:quote ? e.args[2] : replace_syms(e.args[2], membernames))
elseif e.head != :quote
return Expr(e.head, (isempty(e.args) ? e.args : map(x -
replace_syms(x, membernames), e.args))...)
else
if haskey(membernames, e.args[1])
return membernames[e.args[1]]
else
a = gensym()
membernames[e.args[1]] = a
return a
end
end
end
function with_helper(d, body)
membernames = Dict{Symbol, Symbol}()
body = replace_syms(body, membernames)
funargs = map(x - :( getindex($d, $(Meta.quot(x))) ),
collect(keys(membernames)))
funname = gensym()
return(:( function $funname($(collect(values(membernames))...)) $body
end; $funname($(funargs...)) ))
end
macro with(d, body)
esc(with_helper(d, body))
end
On Wednesday, July 8, 2015 at 3:50:12 PM UTC-4, Brandon Taylor wrote:
It should be noted that in R, environments are simply pointers. They take
up no memory and shouldn't cause drastic slowdowns (I don't think).
On Wednesday, July 8, 2015 at 3:48:08 PM UTC-4, Brandon Taylor wrote:
For example, here's a code snippet from R:
function (x, data) eval(x$expr, data, x$env)
x is an expression which contains an environment attribute. It is
evaluated first checking the bindings in data (data being a dataframe) and
then checing the bindings in x's environment.
On Wednesday, July 8, 2015 at 3:37:04 PM UTC-4, Brandon Taylor wrote:
*reifying. Deifying environments might not be the best idea.
On Wednesday, July 8, 2015 at 3:34:53 PM UTC-4, Brandon Taylor wrote:
I was aware of those packages (though I hadn't read the discussions
referenced). Macros are great but they are incredibly difficult to reason
with concerning issues of scope (at least for me). Deifying environments
could solve all of these issues (and so much more) in one fell swoop.
On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote:
Some of these issues have been thought about fairly extensively by
the stats community in particular, precisely on account of the use cases
you cite:
https://github.com/JuliaStats/DataFrames.jl/pull/472
https://github.com/JuliaStats/DataFrames.jl/issues/504
https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FJuliaStats%2FDataFrames.jl%2Fissues%2F504sa=Dsntz=1usg=AFQjCNHgUEZP8TyJ_BuUyyFA5SIxneOJTA
I think that the matter is still very much an open question. I have
no sense that anything is going to be added to Base Julia itself.
Currently, the best way (that I know of, anyway) to achieve the delayed
evaluation effect is via the use of macros. See for instance:
https://github.com/JuliaStats/DataFramesMeta.jl
https://github.com/one-more-minute/Lazy.jl
I'm hope somebody else will be able to pop in an give a more thorough
answer, but the above may at least be a place to start.
On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote:
Hadley Wickham's lazyeval package in R is pretty cool in that you
can attach an environment to an expression, pass it in and out of
functions
with various modifications, and then evaluate the expression within the
original environment (or any other environment that you choose). R in
general has the functions like list2env and list(environment()) that
allow
one to convert an environment into a list and back again (list being
the R
equivalent of a Dict). Are there any plans to add these kind of
features to
Julia?
[julia-users] Re: Environment reification and lazy evaluation
Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
Re: [julia-users] Re: Environment reification and lazy evaluation
Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
[julia-users] Re: Environment reification and lazy evaluation
For comparison, here is the analogous code from DataFramesMeta.
replace_syms(x, membernames) = x
function replace_syms(e::Expr, membernames)
if e.head == :call length(e.args) == 2 e.args[1] == :^
return e.args[2]
elseif e.head == :. # special case for :a.b
return Expr(e.head, replace_syms(e.args[1], membernames),
typeof(e.args[2]) == Expr e.args[2].head ==
:quote ? e.args[2] : replace_syms(e.args[2], membernames))
elseif e.head != :quote
return Expr(e.head, (isempty(e.args) ? e.args : map(x -
replace_syms(x, membernames), e.args))...)
else
if haskey(membernames, e.args[1])
return membernames[e.args[1]]
else
a = gensym()
membernames[e.args[1]] = a
return a
end
end
end
function with_helper(d, body)
membernames = Dict{Symbol, Symbol}()
body = replace_syms(body, membernames)
funargs = map(x - :( getindex($d, $(Meta.quot(x))) ),
collect(keys(membernames)))
funname = gensym()
return(:( function $funname($(collect(values(membernames))...)) $body
end; $funname($(funargs...)) ))
end
macro with(d, body)
esc(with_helper(d, body))
end
On Wednesday, July 8, 2015 at 3:50:12 PM UTC-4, Brandon Taylor wrote:
It should be noted that in R, environments are simply pointers. They take
up no memory and shouldn't cause drastic slowdowns (I don't think).
On Wednesday, July 8, 2015 at 3:48:08 PM UTC-4, Brandon Taylor wrote:
For example, here's a code snippet from R:
function (x, data) eval(x$expr, data, x$env)
x is an expression which contains an environment attribute. It is
evaluated first checking the bindings in data (data being a dataframe) and
then checing the bindings in x's environment.
On Wednesday, July 8, 2015 at 3:37:04 PM UTC-4, Brandon Taylor wrote:
*reifying. Deifying environments might not be the best idea.
On Wednesday, July 8, 2015 at 3:34:53 PM UTC-4, Brandon Taylor wrote:
I was aware of those packages (though I hadn't read the discussions
referenced). Macros are great but they are incredibly difficult to reason
with concerning issues of scope (at least for me). Deifying environments
could solve all of these issues (and so much more) in one fell swoop.
On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote:
Some of these issues have been thought about fairly extensively by the
stats community in particular, precisely on account of the use cases you
cite:
https://github.com/JuliaStats/DataFrames.jl/pull/472
https://github.com/JuliaStats/DataFrames.jl/issues/504
https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FJuliaStats%2FDataFrames.jl%2Fissues%2F504sa=Dsntz=1usg=AFQjCNHgUEZP8TyJ_BuUyyFA5SIxneOJTA
I think that the matter is still very much an open question. I have no
sense that anything is going to be added to Base Julia itself. Currently,
the best way (that I know of, anyway) to achieve the delayed evaluation
effect is via the use of macros. See for instance:
https://github.com/JuliaStats/DataFramesMeta.jl
https://github.com/one-more-minute/Lazy.jl
I'm hope somebody else will be able to pop in an give a more thorough
answer, but the above may at least be a place to start.
On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote:
Hadley Wickham's lazyeval package in R is pretty cool in that you can
attach an environment to an expression, pass it in and out of functions
with various modifications, and then evaluate the expression within the
original environment (or any other environment that you choose). R in
general has the functions like list2env and list(environment()) that
allow
one to convert an environment into a list and back again (list being the
R
equivalent of a Dict). Are there any plans to add these kind of features
to
Julia?
Re: [julia-users] Re: Environment reification and lazy evaluation
All functions. On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
Re: [julia-users] Re: Environment reification and lazy evaluation
On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc1...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon.taylor...@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open
Re: [julia-users] Re: Environment reification and lazy evaluation
They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com javascript: wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an
Re: [julia-users] Re: Environment reification and lazy evaluation
Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with
Re: [julia-users] Re: Environment reification and lazy evaluation
Edit: Hmm, so that means that any implementation of environments would have to be handled in two separate ways: one by the compiler for non-global scope, and one in run time for the special global system?] On Wednesday, July 8, 2015 at 9:49:50 PM UTC-4, Brandon Taylor wrote: Hmm, so that means that any implementation of environments would have to be handled in to separate ways: one by the compiler for non-global scope, and one in run time using a second for the special global system?] On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are
Re: [julia-users] Re: Environment reification and lazy evaluation
There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon.taylor...@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
Re: [julia-users] Re: Environment reification and lazy evaluation
On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon.taylor...@gmail.com wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments of the function/package you called, in your function, or in the global environment. Regards On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4,
Re: [julia-users] Re: Environment reification and lazy evaluation
Hmm, so that means that any implementation of environments would have to be handled in to separate ways: one by the compiler for non-global scope, and one in run time using a second for the special global system?] On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu wrote: On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu yyc...@gmail.com javascript: wrote: On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor brandon@gmail.com javascript: wrote: Hmm, maybe I'm confused about compilation vs interpretation. Let me rephrase. Regardless of a how or when statement is evaluated, it must have access at least to its parent environments to successfully resolve a symbol. AFAIK, the only scope you can dynamically add variable to is the global scope. (This can be done with the `global` keyword or `eval` etc). The compiler figure out the creation/lifetime of all local variables (at compile time). Therefore, to access a variable in the parent scope: 1. If it's a global, then it need a runtime lookup/binding (the reason global are slow) 2. If it's in a parent non-global scope, the compiler can figure out how to bind/access it at compile time and no extra (lookup) code at runtime is necessary. A julia local variable is basically a variable in C. There's a table at compile time to map between symbols and stack slots (or whereever they are stored) but such a map does not exist at runtime anymore (except for debugging). On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon Taylor wrote: They must exist at runtime and at local scope. Evaluating a symbol is impossible without a pool of defined symbols in various scopes to match it to. Unless I'm missing something? On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson wrote: There are global symbol tables for static analysis / reflection, but they do not exist at runtime or for the local scope. On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor brandon@gmail.com wrote: Surely environments already exist somewhere inside Julia? How else could you keep track of scope? It would be simply a matter of granting users access to them. Symbol tables in a mutable language are by default mutable. It would certainly be possible only give users access to immutable reifications (which could solve a bunch of problems as is). However, it seems natural to match mutable symbol tables with mutable reifications, and immutable symbol tables with immutable reifications. On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4, Brandon Taylor wrote: I'm not sure I understand... On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John Myles White wrote: Reified scope makes static analysis much too hard. Take any criticism of mutable state: they all apply to globally mutable symbol tables. On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon Taylor a écrit : All functions. Well, I don't know of any language which doesn't have scoping rules... Anyway, I didn't say scoping rules are necessarily confusing, I was only referring to R formulas. But according to the examples you posted, your question appears to be different. Regards On Wednesday, July 8, 2015 at 4:18:09 PM UTC-4, Milan Bouchet-Valat wrote: Le mercredi 08 juillet 2015 à 12:57 -0700, Brandon Taylor a écrit : If scoping rules are too complicated and cause confusion, why are they built into the base implementation of function? What do you mean? Which function? On Wednesday, July 8, 2015 at 3:48:52 PM UTC-4, Milan Bouchet -Valat wrote: Le mercredi 08 juillet 2015 à 12:34 -0700, Brandon Taylor a écrit : I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On the contrary, I think well-designed macros can be much easier to think about than environments in R. If the macro takes a DataFrame object and an expression, there's no ambiguity about what the scope is. This is even better if variables that should be found in the data frame are passed as symbols, like :var, while standard variables are specified as usual. On the other hand, I find R formulas too flexible and complex to reason about. You never know whether an object will be found in the formula's environment, in one of the parent environments
[julia-users] Re: Environment reification and lazy evaluation
*reifying. Deifying environments might not be the best idea. On Wednesday, July 8, 2015 at 3:34:53 PM UTC-4, Brandon Taylor wrote: I was aware of those packages (though I hadn't read the discussions referenced). Macros are great but they are incredibly difficult to reason with concerning issues of scope (at least for me). Deifying environments could solve all of these issues (and so much more) in one fell swoop. On Wednesday, July 8, 2015 at 3:20:00 PM UTC-4, David Gold wrote: Some of these issues have been thought about fairly extensively by the stats community in particular, precisely on account of the use cases you cite: https://github.com/JuliaStats/DataFrames.jl/pull/472 https://github.com/JuliaStats/DataFrames.jl/issues/504 https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FJuliaStats%2FDataFrames.jl%2Fissues%2F504sa=Dsntz=1usg=AFQjCNHgUEZP8TyJ_BuUyyFA5SIxneOJTA I think that the matter is still very much an open question. I have no sense that anything is going to be added to Base Julia itself. Currently, the best way (that I know of, anyway) to achieve the delayed evaluation effect is via the use of macros. See for instance: https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/one-more-minute/Lazy.jl I'm hope somebody else will be able to pop in an give a more thorough answer, but the above may at least be a place to start. On Wednesday, July 8, 2015 at 2:03:45 PM UTC-4, Brandon Taylor wrote: Hadley Wickham's lazyeval package in R is pretty cool in that you can attach an environment to an expression, pass it in and out of functions with various modifications, and then evaluate the expression within the original environment (or any other environment that you choose). R in general has the functions like list2env and list(environment()) that allow one to convert an environment into a list and back again (list being the R equivalent of a Dict). Are there any plans to add these kind of features to Julia?
