Re: [julia-users] DataFrame Manipulation (like dplyr) Help

[Tom Short]

The DataFramesMeta.jl package has one approach. Two new kids on the block:
jplyr.jl and Query.jl have different approaches for that type of function.

On Aug 26, 2016 1:38 PM,  wrote:

Hi,

I'm exited about Julia.  Coming from an R and SAS background I am trying to
develop some familiar tools to manipulate and query data.  So far here are
some of the things I have, but I just want feedback to see if there is a
better way.  The following code attempts to create a summarize() function
similar to that of dplyr in R.  But the call is a bit odd to look at (see
below).  Is there something I can do to avoid using :() or quotes?

# function definitions are below

using DataFrames

dat = DataFrame(x=rand(10), y=rand(10))
summary = summarize(dat, mu_x = :(mean(:x)), mu_y = :(mean(:y)));


# Function definitions manipulate expressions (Expr types) to
# access data frame elements

function df_statement(df, u::Expr)

  t = u
  arg_id = 1

  for ar in t.args
if typeof(ar) == QuoteNode
  t.args[arg_id] = df[eval(ar)]
elseif typeof(ar) == Expr
  df_statement(df, ar)
end
arg_id += 1
  end

  return eval(t)

end


function summarize(df; kwargs...)

  df_out = DataFrame()

  for (key, value) in kwargs
  df_out[Symbol(key)] = df_statement(df, value)
  end

  return df_out

end

[julia-users] [ANN] Calc.jl - a calculator for the REPL

[Tom Short]

https://github.com/tshort/Calc.jl

This adds a REPL mode for an RPN calculator initiated by the = key.
Operations and key strokes generally mimic Emacs Calc. It's pretty easy to
add your own operations. The RPN style can use less keys. Undo and redo are
easy.

Re: [julia-users] How To Use Julia with SAP via NetWeaver RFC Library

[Tom Short]

Looks like something that could go into a package. If you don't want to do
that, it'd be nice if you could add a license to your code, so others could
use it as a starting point.
On Jul 10, 2016 4:45 PM, "Stefan Schnell"  wrote:

> Hello community,
>
> here an include how to use Julia with SAP via NetWeaver RFC library - but
> only with a few functions, not the complete set of the SAP NetWeaver RFC
> library:
>
> #-Begin-
>
>   #-User Defined Types--
> type RFC_CONNECTION_PARAMETER
>   nameASHost::Ptr{UInt16}
>   valueASHost::Ptr{UInt16}
>   nameSysNr::Ptr{UInt16}
>   valueSysNr::Ptr{UInt16}
>   nameClient::Ptr{UInt16}
>   valueClient::Ptr{UInt16}
>   nameUser::Ptr{UInt16}
>   valueUser::Ptr{UInt16}
>   namePassWd::Ptr{UInt16}
>   valuePassWd::Ptr{UInt16}
>   nameLang::Ptr{UInt16}
>   valueLang::Ptr{UInt16}
> end
>
> type RFC_ERROR_INFO
>   code::Int32
>   group::Int32
>   key::Ptr{UInt16}
>   message::Ptr{UInt16}
>   abapMsgClass::Ptr{UInt16}
>   abapMsgType::Ptr{UInt16}
>   abapMsgNumber::Ptr{UInt16}
>   abapMsgV1::Ptr{UInt16}
>   abapMsgV2::Ptr{UInt16}
>   abapMsgV3::Ptr{UInt16}
>   abapMsgV4::Ptr{UInt16}
> end
>
>   #-Constants---
> const sapnwrfc = "sapnwrfc.dll"
>
>   #-Main
> cd("C:\\Projects\\SAP\\RFC\\Julia")
>
> code = 0
> group = 0
> key = ""
> message = ""
> abapMsgClass = ""
> abapMsgType = ""
> abapMsgNumber = ""
> abapMsgV1 = ""
> abapMsgV2 = ""
> abapMsgV3 = ""
> abapMsgV4 = ""
> RfcErrorInfo = RFC_ERROR_INFO(
>  code, group, pointer(utf16(key)), pointer(utf16(message)),
>  pointer(utf16(abapMsgClass)), pointer(utf16(abapMsgType)),
>  pointer(utf16(abapMsgNumber)), pointer(utf16(abapMsgV1)),
>  pointer(utf16(abapMsgV2)), pointer(utf16(abapMsgV3)),
>  pointer(utf16(abapMsgV4))
> )
>
>   #-RfcGetVersion---
> function RfcGetVersion()
>   majorVersion = Ref{UInt32}(0)
>   minorVersion = Ref{UInt32}(0)
>   patchLevel = Ref{UInt32}(0)
>   ptrVersion = ccall((:RfcGetVersion, sapnwrfc), stdcall,
> Ptr{UInt16}, (Ptr{UInt32}, Ptr{UInt32}, Ptr{UInt32}),
> majorVersion, minorVersion, patchLevel)
>   return utf16(ptrVersion), majorVersion[], minorVersion[],
> patchLevel[]
> end
>
>   #-RfcOpenConnection---
> function RfcOpenConnection(ashost::AbstractString,
>   sysnr::AbstractString, client::AbstractString,
>   user::AbstractString, passwd::AbstractString,
>   lang::AbstractString = "EN")
>   nameASHost = "ASHOST"
>   valueASHost = ashost
>   nameSysNr = "SYSNR"
>   valueSysNr = sysnr
>   nameClient = "CLIENT"
>   valueClient = client
>   nameUser = "USER"
>   valueUser = user
>   namePassWd = "PASSWD"
>   valuePassWd = passwd
>   nameLang = "LANG"
>   valueLang = lang
>   RfcConnectionParameter = RFC_CONNECTION_PARAMETER(
> pointer(utf16(nameASHost)), pointer(utf16(valueASHost)),
> pointer(utf16(nameSysNr)), pointer(utf16(valueSysNr)),
> pointer(utf16(nameClient)), pointer(utf16(valueClient)),
> pointer(utf16(nameUser)), pointer(utf16(valueUser)),
> pointer(utf16(namePassWd)), pointer(utf16(valuePassWd)),
> pointer(utf16(nameLang)), pointer(utf16(valueLang))
>   )
>   return ccall((:RfcOpenConnection, sapnwrfc), stdcall,
> UInt32, (Ref{RFC_CONNECTION_PARAMETER}, UInt32,
> Ref{RFC_ERROR_INFO}), RfcConnectionParameter, 6, RfcErrorInfo)
> end
>
>   #-RfcGetFunctionDesc--
> function RfcGetFunctionDesc(rfcHandle::UInt32,
>   funcName::AbstractString)
>   return ccall((:RfcGetFunctionDesc, sapnwrfc), stdcall,
> UInt32, (UInt32, Ptr{UInt16}, Ref{RFC_ERROR_INFO}),
> rfcHandle, utf16(funcName), RfcErrorInfo)
> end
>
>   #-RfcCreateFunction---
> function RfcCreateFunction(funcDescHandle::UInt32)
>   return ccall((:RfcCreateFunction, sapnwrfc), stdcall,
> UInt32, (UInt32, Ref{RFC_ERROR_INFO}),
> funcDescHandle, RfcErrorInfo)
> end
>
>   #-RfcInvoke---
> function RfcInvoke(rfcHandle::UInt32, funcHandle::UInt32)
>   return ccall((:RfcInvoke, sapnwrfc), stdcall,
> UInt32, (UInt32, UInt32, Ref{RFC_ERROR_INFO}),
> rfcHandle, funcHandle, RfcErrorInfo)
> end
>
>   #-RfcDestroyFunction--
> function 

Re: [julia-users] Why does julia use END for block end?

[Tom Short]

You can make a quick macro for one-line if statements:

macro when(condition, expr)
esc(:( if $condition; $expr; end))
end

@when 4 > pi  x = 2

For even more compact syntax, you can replace `when` with `?`.




On Fri, May 6, 2016 at 3:12 PM, Adrian Salceanu 
wrote:

> The only place where I find the "end" requirement annoying is for one line
> IF statements. When you have a short one liner, the "end" part just does
> not feel right. It would be nice if the "end" could be left out for one
> liners. Even PHP allows one to skip the accolades in such cases.
>
> If there's some other way of achieving this I'd love to hear about it. I
> don't like the ternary operator in this situation cause it forces me to add
> the 3rd part as "nothing" or whatever. And doing "expr1 && expr2" only
> works when expr2 is "return" for instance, otherwise the compiler complains
> about using a non-boolean in a boolean context.
>
>
> vineri, 6 mai 2016, 20:37:49 UTC+2, Stefan Karpinski a scris:
>>
>> There is a long history of languages using this syntax, including Algol,
>> Pascal, Ruby and Matlab.
>>
>> On Fri, May 6, 2016 at 2:26 PM, Ford Ox  wrote:
>>
>>> Is there any reasoning behind it? It seems to me like a weird choice
>>> since you have to type three letters, which is the complete opposite of the
>>> goal of this language - being very productive (a lot work done with little
>>> code).
>>> On top of that, brain has to read the word every time your eyes look at
>>> it so you spend more time also reading the code - tho this should be easy
>>> to omit, by highlighting this keyword by other color than other keywords
>>> (the current purple color in ATOM just drives me crazy, since it is one of
>>> the most violent colors, so my eyes always try to read that useless piece
>>> of information first, instead of the important code).
>>>
>>
>>

Re: [julia-users] Re: Reproducible research for Julia

[Tom Short]

Daniel's pandoc system is here:

https://github.com/dcjones/Judo.jl



On Thu, May 5, 2016 at 10:04 AM, kleinsplash 
wrote:

>
> Is there an update on these thoughts? Because that link doesnt work
> anymore..
> On Thursday, 9 May 2013 16:35:39 UTC+2, Stephen Eglen wrote:
>>
>> Hi,
>>
>> Coming from R, I've got used to the great Sweave/knitr system for
>> combining latex and R code in the same document, using noweb markup.  Is
>> there anything similar yet for Julia?  Or is there something similar using
>> markdown to embed Julia code into documentation?
>>
>> Thanks, Stephen
>>
>

Re: [julia-users] Re: Newbie question. Need help with grouping dataframes, cumulative sums and plotting.

[Tom Short]

The closest I could come to your pandas approach is with the following.
DataFrames don't have an index, but GroupedDataFrames do. I created an
`ave` function that does something like the R function with the same name:

using DataFrames, DataFramesMeta, RDatasets
df = dataset("datasets", "iris")
df = df[shuffle(vcat(1:nrow(df))),:]

ave(g::GroupedDataFrame, col::Union{Symbol, Int}, fun::Function) =
ave(g, g.parent[col], fun)

function ave(g::GroupedDataFrame, x::AbstractArray, fun::Function)
idx = sub(g.idx, g.starts[1]:g.ends[1])
first = fun(sub(x, idx))
res = similar(first, nrow(g.parent))
res[idx] = first
for i in 2:length(g.starts)
idx = sub(g.idx, g.starts[i]:g.ends[i])
res[idx] = fun(sub(x, idx))
end
res
end

df[:cs] = ave(groupby(df, :Species), :PetalLength, cumsum)

​


On Wed, May 4, 2016 at 9:23 AM, Cedric St-Jean 
wrote:

> That's way better, thank you!
>
> I never thought I'd say this, but I miss pandas. I could write
>
> df['cs'] = df.groupby('PetalLength').transform(cumsum)
>
> That's not possible in Julia because DataFrames don't have a row index.
>
> On Wednesday, May 4, 2016 at 9:04:21 AM UTC-4, tshort wrote:
>>
>> Here's another way with DataFramesMeta [1]:
>>
>> using DataFrames, DataFramesMeta, RDatasets
>> df = dataset("datasets", "iris")@transform(groupby(df, :Species), cs = 
>> cumsum(:PetalLength))
>>
>> ​
>>
>> [1] https://github.com/JuliaStats/DataFramesMeta.jl/
>>
>>
>>
>>
>> On Wed, May 4, 2016 at 8:09 AM, Cedric St-Jean 
>> wrote:
>>
>>> "Do blocks" are one of my favourite things about Julia, they're
>>> explained in the docs
>>> .
>>> Basically it's just a convenient way of defining and passing a function
>>> (the code that comes after `do`) to another function (in this case, `by`).
>>> `by` goes over the dataframe, splits it into 3 subdataframes (one for each
>>> Species in the iris dataset), and calls the do-block for each of them. Then
>>> their return values (the last line in the do-block) gets concatenated
>>> together to form the final result. The code I really wanted to write is:
>>>
>>> using RDatasets
>>> df = dataset("datasets", "iris")
>>> # For each species
>>> df2 = by(df, :Species) do sub_df
>>>sub_df = copy(sub_df)   # don't modify the original dataframe
>>># Add a :cumulative_PetalLength column
>>>sub_df[:cumulative_PetalLength] = cumsum(sub_df[:PetalLength])
>>># Return the new sub-dataframe
>>>   sub_df
>>> end
>>>
>>> but unfortunately, this code doesn't work with DataFrames.jl
>>>
>>>
>>> On Wednesday, May 4, 2016 at 4:42:41 AM UTC-4, Ben Southwood wrote:

 Thanks Cedric, that worked very well.  I'm having a little trouble
 following the documentation as to how the "by ... do ..." structure
 actually works.  Would you mind explaining what the code is doing?

 On Tuesday, May 3, 2016 at 10:07:10 PM UTC-4, Cedric St-Jean wrote:
>
> Something like
>
> using RDatasets
> df = dataset("datasets", "iris")
> df[:cumulative_PetalLength] = 0.0
> by(df, :Species) do sub_df
> sub_df[:cumulative_PetalLength] = cumsum(sub_df[:PetalLength])
> sub_df
> end
>
> though I hope someone can provide a more elegant solution. `sub_df` a
> SubDataFrame, and those objects can neither have a new column nor be
> converted to DataFrame.
>
> On Tuesday, May 3, 2016 at 4:22:29 PM UTC-4, Ben Southwood wrote:
>>
>> I have the following dataframe with values of the form
>>
>> date1,label1,qty1_1
>> date2,label1,qty1_2
>> date3,label1,qty1_3
>> 
>> dateN,label1,qty1_N
>> date1,label2,qty2_1
>> date2,label2,qty2_2
>> date3,label2,qty2_3
>> 
>> dateN,label2,qty1_N
>> 
>>
>>
>>
>> I would like to cumulative sum the qtys such that the value of the
>> cumulative sum only increases for each label. And then i'd have
>>
>> date1,label1,cuml1_1
>> date2,label1,cuml1_2
>> date3,label1,cuml1_3
>> 
>> dateN,label1,cuml1_N
>> date1,label2,cuml2_1
>>
>>
>>
>> This way I can use gadfly and run the following plot
>>
>>
>> plot(x=grouped[:date],y=grouped[:cuml_sum],color=grouped[:label],Geom.line)
>>
>>
>> and have each cuml sum have it's own colouring by date.  I'm stuck on
>> how to do this simply without creating lookups. Any help? Thanks!
>>
>>
>>
>>

Re: [julia-users] Re: Newbie question. Need help with grouping dataframes, cumulative sums and plotting.

[Tom Short]

Here's another way with DataFramesMeta [1]:

using DataFrames, DataFramesMeta, RDatasets
df = dataset("datasets", "iris")@transform(groupby(df, :Species), cs =
cumsum(:PetalLength))

​

[1] https://github.com/JuliaStats/DataFramesMeta.jl/



On Wed, May 4, 2016 at 8:09 AM, Cedric St-Jean 
wrote:

> "Do blocks" are one of my favourite things about Julia, they're explained in
> the docs
> .
> Basically it's just a convenient way of defining and passing a function
> (the code that comes after `do`) to another function (in this case, `by`).
> `by` goes over the dataframe, splits it into 3 subdataframes (one for each
> Species in the iris dataset), and calls the do-block for each of them. Then
> their return values (the last line in the do-block) gets concatenated
> together to form the final result. The code I really wanted to write is:
>
> using RDatasets
> df = dataset("datasets", "iris")
> # For each species
> df2 = by(df, :Species) do sub_df
>sub_df = copy(sub_df)   # don't modify the original dataframe
># Add a :cumulative_PetalLength column
>sub_df[:cumulative_PetalLength] = cumsum(sub_df[:PetalLength])
># Return the new sub-dataframe
>   sub_df
> end
>
> but unfortunately, this code doesn't work with DataFrames.jl
>
>
> On Wednesday, May 4, 2016 at 4:42:41 AM UTC-4, Ben Southwood wrote:
>>
>> Thanks Cedric, that worked very well.  I'm having a little trouble
>> following the documentation as to how the "by ... do ..." structure
>> actually works.  Would you mind explaining what the code is doing?
>>
>> On Tuesday, May 3, 2016 at 10:07:10 PM UTC-4, Cedric St-Jean wrote:
>>>
>>> Something like
>>>
>>> using RDatasets
>>> df = dataset("datasets", "iris")
>>> df[:cumulative_PetalLength] = 0.0
>>> by(df, :Species) do sub_df
>>> sub_df[:cumulative_PetalLength] = cumsum(sub_df[:PetalLength])
>>> sub_df
>>> end
>>>
>>> though I hope someone can provide a more elegant solution. `sub_df` a
>>> SubDataFrame, and those objects can neither have a new column nor be
>>> converted to DataFrame.
>>>
>>> On Tuesday, May 3, 2016 at 4:22:29 PM UTC-4, Ben Southwood wrote:

 I have the following dataframe with values of the form

 date1,label1,qty1_1
 date2,label1,qty1_2
 date3,label1,qty1_3
 
 dateN,label1,qty1_N
 date1,label2,qty2_1
 date2,label2,qty2_2
 date3,label2,qty2_3
 
 dateN,label2,qty1_N
 



 I would like to cumulative sum the qtys such that the value of the
 cumulative sum only increases for each label. And then i'd have

 date1,label1,cuml1_1
 date2,label1,cuml1_2
 date3,label1,cuml1_3
 
 dateN,label1,cuml1_N
 date1,label2,cuml2_1



 This way I can use gadfly and run the following plot


 plot(x=grouped[:date],y=grouped[:cuml_sum],color=grouped[:label],Geom.line)


 and have each cuml sum have it's own colouring by date.  I'm stuck on
 how to do this simply without creating lookups. Any help? Thanks!

Re: [julia-users] Time handling: Micro/Nanosecond precision and time string formatting like Unix

[Tom Short]

It's not that hard to extend the date handling in Base to handle this.
Here's one example with microsecond accuracy:

https://github.com/tshort/COMTRADE.jl/blob/master/src/datetime.jl

I also like the idea of a TimeOfDay type like Tom B has implemented.



On Fri, Apr 29, 2016 at 8:06 AM, Ben Southwood 
wrote:

> Are there any packages that can handle "Unix style" times?  How come Julia
> can only handle seconds in 0.4.5 and milliseconds in 0.5 (unstable)?
> Shouldn't we just aim big and go all the way to nanos?
>
> For example, it would be great if I could handle the following times.
>
> 2015-12-11 09:46:40.882362Z
>
> 2015-09-11 14:37:12.960014+01:00,
>
>

Re: [julia-users] Any downside of using a symbol as an interned string?

[Tom Short]

I've used symbols as strings before, and it's worked fine.

I've also developed an unregistered package to pool elements. It includes a
PooledString and PooledStringArray. My main use was for DataFrames. By
pooling elements, particularly strings, you can group and sort much faster.
Strings are mapped to integers. For this use case, symbols did not work as
well. You might find that useful to use or fork.

https://github.com/tshort/PooledElements.jl

Here's background from the DataFrames perspective:

https://github.com/JuliaStats/DataFrames.jl/issues/895

Hopefully, I'll find the time and ambition to better integrate this package
with DataFrames.



On Fri, Apr 22, 2016 at 7:56 AM, Lyndon White  wrote:

> @Tamas thanks
> Can you clarify a Dict of strings to what? Ints?
> Loose a lot of interpretablity that way.
>
>
> On Friday, 22 April 2016 19:08:11 UTC+8, Tamas Papp wrote:
>>
>> I would be more concerned about style than speed -- symbols as strings
>> is an ancient Lisp technique in NLP, but IMO a Dict of strings would be
>> better style.
>>
>>
>>

Re: [julia-users] [ANN] FixedSizeDictionaries

[Tom Short]

Great, Simon! How do you feel this compares with the following?

https://github.com/blackrock/NamedTuples.jl

One difference is using generated functions versus on-the-fly type
creation. Any trade-offs in performance or usability from that?


On Mon, Apr 11, 2016 at 8:08 AM, Simon Danisch  wrote:

> Here is yet another package: FixedSizeDictionaries.jl
> 
>
> From the README:
> *Library which implements a FixedSize variant of Dictionaries. These can
> be stack allocated and have O(1) indexing performance without boundcheck.
> It implements most parts of the Base.Dict interface. *
> *This package is useful, when you want anonymous composite types. You
> should be a bit careful with generating a lot of FixedSizeDict's, since it
> will compile a unique set of functions for every field of a Dict.*
>
> I'll be using it to speed up various places where I'm currently using a
> dictionary but where the number of keys is set at compile time.
> Also, Shashi  and I contemplated to implement
> Extensible Records
> , which seems like a good fit to
> represent all kind of graphics/geometry types.
> FixedSizeDictionaries could be an integral part of this!
>
> Best,
> Simon
>
> PS:
> I do feel like it's time for a package announcement list...
>

Re: [julia-users] How to copy a DataFrame structure in an empty DataFrame

[Tom Short]

Feel free to file issues for missing documentation or (even better) pull
requests.

On Thu, Mar 24, 2016 at 2:24 PM, Fred  wrote:

> Thank you Milan for this useful link.
> This website contains more informations than mine, but it does not seems
> to contains all the functions available for dataframes. For example I did
> not manage to find that I have to use append! instead of push! to add a row
> to a dataframe.
>

Re: [julia-users] Re: dropna for DataFrames

[Tom Short]

See complete_cases and complete_cases!.
On Mar 17, 2016 10:37 AM, "Cedric St-Jean"  wrote:

> I use this function:
>
> """ `dropnan_dumb(df::DataFrame, axis=1)`
>
> Returns `df` with all rows that have any `NA` removed """
> function dropnan_dumb(df::DataFrame, axis=1)
> @assert axis==1 # TODO
> todrop = fill(false, size(df, 1))
> for col in names(df)
> todrop = todrop | [isna(el) for el in df[col]]
> end
> df[find(~todrop), :]
> end
>
>
>
> On Thursday, March 17, 2016 at 10:17:36 AM UTC-4, Eugen Neu wrote:
>>
>>
>> Hi,
>>
>> I am a new julia user. I have a dataset (
>> http://www-bcf.usc.edu/~gareth/ISL/Auto.csv) where one column contains
>> NAs. The data can be read using:
>>
>> Auto = readtable("data/Auto.csv", nastrings = ["?"])
>>
>> I would like to remove the rows that contain NAs. I found the dropna
>> function. However, this seems to work only for DataArrays not for
>> DataFrames. I also found:
>> http://stackoverflow.com/questions/27844621/how-can-i-delete-all-rows-of-a-dataframe-that-have-an-na-in-a-specific-column
>> However, this only works when one specifies the column that contains NAs.
>>
>> Is there a function that removes every DataFrame row that includes one or
>> more NAs?
>>
>> Kind regards,
>> Eugen
>>
>

Re: [julia-users] Re: Mmap allocation question

[Tom Short]

Where does your variable `dims` come from? As pointed out above, global
variables can hurt type inference.
On Mar 13, 2016 8:29 AM, "Tim Loderhose"  wrote:

> I implemented the suggestions (see updated gist:
> https://gist.github.com/timlod/0f607e311d0464fd6c63).
> The allocation in the for loops disappeared and the time required halved
> to ~3s. The mmap access though still allocated a lot.
> And the vectorised access is still much faster and allocates less
> (although I want to get rid of the allocation altogether).
> Any other ideas?
>
> On Saturday, 12 March 2016 15:15:51 UTC+1, Dan wrote:
>>
>> Yep, `peCounters`, `paCounters` and `dims` are not type-stable. They are
>> one type by their default values and then assigned another. Perhaps rename
>> the default parameters, and copy them to `peCounters`, `paCounters` and
>> `dims` only if they are set to something other than `0`.
>>
>> Also, `mdhcounters` might not return a definite type (need to check that
>> function).
>> Fixing these should make the loop efficient.
>>
>> On Saturday, March 12, 2016 at 4:05:39 PM UTC+2, Tim Loderhose wrote:
>>>
>>> Here's the actual code:
>>> https://gist.github.com/timlod/0f607e311d0464fd6c63
>>> 
>>>
>>> I am running the code from the REPL, may that be a problem? (As I read
>>> in the REPL everything is global). In the file nothing is global.
>>> Also, the counters are UInt16s, but that shouldnt matter I guess.
>>>
>>> Thanks for the help so far!
>>>
>>> On Saturday, 12 March 2016 14:22:38 UTC+1, Dan wrote:

 It's better to have code which actually runs in the post. In any case,
 the allocations at the `for` lines is suspicious - the for should basically
 only allocate a counter. Are there any global variables? Is `counter1` or
 `counter2` or `dims` global? Globals are always a good source of confusion
 to the type-inference engine.

 On Saturday, March 12, 2016 at 2:28:51 PM UTC+2, Tim Loderhose wrote:
>
> The code is in a function. I changed the names a bit to make it more
> understandable. The actual function is longer and has different variable
> names.
>
> On Saturday, 12 March 2016 13:01:28 UTC+1, tshort wrote:
>>
>> Is that code in a function? (It should be.) Also, one of your
>> variable names changed to `counter1s`. Suspect a type instability.
>> On Mar 12, 2016 4:12 AM, "Tim Loderhose"  wrote:
>>
>>> I tried around with that a bit, but then it gets much worse: From
>>> ~1s to ~6s, allocation as shown:
>>>
>>> 153710487 mat = Array{Complex64}(dims...)
>>>   4722450   file = Mmap.mmap(filename, Array{Complex64,2},
>>> (dims[2],length(counter1)))
>>>  9568  for i = 1:dims[2]
>>>  4000 for j = 1:length(counter1)
>>> 1690462534  mat[counter1s[j],i,counter2[j]] = file[i,j]
>>> - end
>>>
>>> I swapped the for loops around here, but that didn't matter. I can
>>> gain a little bit by indexing i into the first dimension of mat, but it
>>> still lags far behind.
>>> Any other ideas?
>>>
>>> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote:

 I think array slices (on right hand side of assignment) create new
 arrays, hence the allocation.
 Try writing an explicit loop instead, something like:

 for j = 1:length(counter1)
for i = 1:size(file,1)
mat[counter1[j],i,counter2[j]] = file[i,j]
end
 end


 On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose
 wrote:

> Hi,
>
> I have a question regarding some allocation in my code I would
> like to get rid of.
> I am memory mapping a file (which could be very large) which is
> part of a complex 3D matrix, and then put its contents into the
> preallocated matrix along the second dimension. I need the counters 
> because
> the contents of file are only a subset of the full matrix.
>
> Here's a profiled snippet, where the file which is loaded has
> 120619520 bytes.
>
> 153705063 mat = Array{Complex64}(dims...)
>  4721282file = Mmap.mmap(filename, Array{Complex64,2},
> (dims[2],length(counter1)))
> 16   for i = 1:length(counter1)
> 148179531   mat[counter1[i],:,counter2[i]] = file[:,i]
> -  end
>
> Why does the code allocate so much memory inside the for-loop
> (even more bytes than the contents of file)?
> It seems like this is a trivial matter, right now I just can't get
> my head around it, any help is 

Re: [julia-users] Re: Mmap allocation question

[Tom Short]

Is that code in a function? (It should be.) Also, one of your variable
names changed to `counter1s`. Suspect a type instability.
On Mar 12, 2016 4:12 AM, "Tim Loderhose"  wrote:

> I tried around with that a bit, but then it gets much worse: From ~1s to
> ~6s, allocation as shown:
>
> 153710487 mat = Array{Complex64}(dims...)
>   4722450   file = Mmap.mmap(filename, Array{Complex64,2},
> (dims[2],length(counter1)))
>  9568  for i = 1:dims[2]
>  4000 for j = 1:length(counter1)
> 1690462534  mat[counter1s[j],i,counter2[j]] = file[i,j]
> - end
>
> I swapped the for loops around here, but that didn't matter. I can gain a
> little bit by indexing i into the first dimension of mat, but it still lags
> far behind.
> Any other ideas?
>
> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote:
>>
>> I think array slices (on right hand side of assignment) create new
>> arrays, hence the allocation.
>> Try writing an explicit loop instead, something like:
>>
>> for j = 1:length(counter1)
>>for i = 1:size(file,1)
>>mat[counter1[j],i,counter2[j]] = file[i,j]
>>end
>> end
>>
>>
>> On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose wrote:
>>
>>> Hi,
>>>
>>> I have a question regarding some allocation in my code I would like to
>>> get rid of.
>>> I am memory mapping a file (which could be very large) which is part of
>>> a complex 3D matrix, and then put its contents into the preallocated matrix
>>> along the second dimension. I need the counters because the contents of
>>> file are only a subset of the full matrix.
>>>
>>> Here's a profiled snippet, where the file which is loaded has 120619520
>>> bytes.
>>>
>>> 153705063 mat = Array{Complex64}(dims...)
>>>  4721282file = Mmap.mmap(filename, Array{Complex64,2},
>>> (dims[2],length(counter1)))
>>> 16   for i = 1:length(counter1)
>>> 148179531   mat[counter1[i],:,counter2[i]] = file[:,i]
>>> -  end
>>>
>>> Why does the code allocate so much memory inside the for-loop (even more
>>> bytes than the contents of file)?
>>> It seems like this is a trivial matter, right now I just can't get my
>>> head around it, any help is appreciated :)
>>>
>>> Thanks,
>>> Tim
>>>
>>

Re: [julia-users] Re: [ANN] Maker.jl: a make-like system for data analysis

[Tom Short]

Maybe. I haven't used BinDeps, so I'm not that familiar with it. After
looking at it some, there are parallels given that both provide a
declarative way to manage dependencies. It's not a foreseeable need for me,
though.


On Thu, Mar 10, 2016 at 11:15 PM, Jameson  wrote:

> Could this be used to replace BinDeps? I note that you mention that was
> your primary target, but I'm curious whether you think it would be feasible?
>
>
>
> On Monday, March 7, 2016 at 8:11:30 PM UTC-5, tshort wrote:
>>
>> The following package is now registered.
>>
>> https://github.com/tshort/Maker.jl
>>
>> This package helps manage actions and account for dependencies. It's like
>> *make* or Ruby's Rake, but targeted at data processing and analysis
>> rather than building code. Maker provides methods for defining tasks and
>> dependencies between tasks.
>>
>> My target use case is to define several tasks and dependencies, and run
>> `make()` to perform the analysis. Then if input files change or I change
>> part of the code, run `make()` again, and Maker will re-run only the parts
>> that have changed.
>>
>> Maker was derived from Mike Nolta's Jake.jl. The syntax and API are
>> similar to Rake. Comments welcome.
>>
>>

Re: [julia-users] Retrieving the docstring

[Tom Short]

Here's one way:

Base.Markdown.plain(@doc(length))

On Mon, Mar 7, 2016 at 8:52 PM, Cedric St-Jean 
wrote:

> Is there any way to retrieve the docstring(s) associated to a function as
> a string (not with the help system)? Something like `get_docstring(length)`?
>

[julia-users] [ANN] Maker.jl: a make-like system for data analysis

[Tom Short]

The following package is now registered.

https://github.com/tshort/Maker.jl

This package helps manage actions and account for dependencies. It's like
*make* or Ruby's Rake, but targeted at data processing and analysis rather
than building code. Maker provides methods for defining tasks and
dependencies between tasks.

My target use case is to define several tasks and dependencies, and run
`make()` to perform the analysis. Then if input files change or I change
part of the code, run `make()` again, and Maker will re-run only the parts
that have changed.

Maker was derived from Mike Nolta's Jake.jl. The syntax and API are similar
to Rake. Comments welcome.

Re: [julia-users] Data Structure / Pair question

[Tom Short]

There are several options to "keep things together", particularly with
vectors of the same type:

- DataFrame columns -- watch how you use columns to keep type stability

- Nx2 Array

- Nx2 NamedArray:
https://github.com/davidavdav/NamedArrays.jl

- AxisArrays:
https://github.com/mbauman/AxisArrays.jl


On Mon, Mar 7, 2016 at 5:11 PM, Christopher Alexander 
wrote:

> Yea, I was thinking about two different vectors, but then if I did any
> sorting, the value vector and weight vector would be out-of-sync.  I'll
> check out this StructsOfArrays package
>
> Thanks!
>
> Chris
>
> On Monday, March 7, 2016 at 5:03:14 PM UTC-5, tshort wrote:
>>
>> It depends on what "various weighted statistical calculations" involves.
>> I'd start with two vectors, `x` and `w`. If you really need them to be
>> coupled tightly, you could define an immutable type to hold the value and
>> the weight, but the two separate vectors can be faster for some operations.
>> Also, see:
>>
>> https://github.com/simonster/StructsOfArrays.jl
>>
>> On Mon, Mar 7, 2016 at 4:50 PM, Christopher Alexander 
>> wrote:
>>
>>> Hello all, I need to create a structure where I keep track of pairs of
>>> value => weight so that I can do various weighted statistical calculations.
>>>
>>> I know that StatsBase has a weights vector, which I plan on using, but
>>> the way that is set up is that it is disassociated from each of the values
>>> to which the weights are to be applied.
>>>
>>> I need the mapping that "Pair" provides, but I've noticed that there is
>>> no easy way, if I have an array of pairs, to grab all the first values or
>>> all the second values (like you can do with a dict in grabbing keys or
>>> values).
>>>
>>> I've tried to do something like map(first, my_array_of_pairs), but this
>>> is about 10x slower than if you have a dictionary of value => weight and
>>> just asked for the keys.  I actually tried to use a dict at first, but ran
>>> into issues with duplicate values (they were overwriting each other because
>>> the value was the key).
>>>
>>> Any suggestions, or any better way to manipulate an array of Pairs?
>>>
>>> Thanks!
>>>
>>> Chris
>>>
>>
>>

Re: [julia-users] Data Structure / Pair question

[Tom Short]

It depends on what "various weighted statistical calculations" involves.
I'd start with two vectors, `x` and `w`. If you really need them to be
coupled tightly, you could define an immutable type to hold the value and
the weight, but the two separate vectors can be faster for some operations.
Also, see:

https://github.com/simonster/StructsOfArrays.jl

On Mon, Mar 7, 2016 at 4:50 PM, Christopher Alexander 
wrote:

> Hello all, I need to create a structure where I keep track of pairs of
> value => weight so that I can do various weighted statistical calculations.
>
> I know that StatsBase has a weights vector, which I plan on using, but the
> way that is set up is that it is disassociated from each of the values to
> which the weights are to be applied.
>
> I need the mapping that "Pair" provides, but I've noticed that there is no
> easy way, if I have an array of pairs, to grab all the first values or all
> the second values (like you can do with a dict in grabbing keys or values).
>
> I've tried to do something like map(first, my_array_of_pairs), but this is
> about 10x slower than if you have a dictionary of value => weight and just
> asked for the keys.  I actually tried to use a dict at first, but ran into
> issues with duplicate values (they were overwriting each other because the
> value was the key).
>
> Any suggestions, or any better way to manipulate an array of Pairs?
>
> Thanks!
>
> Chris
>

[julia-users] Re: make() in Julia for data analysis

[Tom Short]

This repo is now at:

https://github.com/tshort/Maker.jl

It should be registered soon.

On Wed, Feb 24, 2016 at 4:22 PM, Tom Short <tshort.rli...@gmail.com> wrote:

> I'm interested in feedback on the following unregistered package that
> implements a build-dependency system like the *make* utility or Ruby's
> Rake:
>
> https://github.com/tshort/Make.jl  (based on Mike Nolta's Rake.jl)
>
> For me, a typical project will have a script that reads in data followed
> by several steps of data processing, analysis, and/or simulation. I
> normally split this into several functions. The main problem with this
> approach is that I typically have to manually run one or more of these
> functions to load variables and do initial processing. Sometimes, I will
> script this setup, but it's still klunky.
>
> The package above tries to help manage these dependencies. In theory, I
> can include my Julia script and run `make()`. This will load variables and
> re-run parts of the data analysis if the input data has changed or I've
> change parts of my code.
>
> Suggestions? Other approaches to handle this use case?
>
>

Re: [julia-users] Comparing functions

[Tom Short]

I'm interested in something like that, too. I only want to rerun parts of
calculations if functions change or if input data has changed. Here's what
I came up with to check functions:

https://github.com/tshort/Make.jl/blob/master/src/Make.jl#L43-L49

It's probably not right but it's working reasonably well for me, at least
for anonymous functions. Changes in Julia v0.5 will probably change this.
I'm currently wrestling with how to maintain some state between Julia
sessions.




On Fri, Feb 26, 2016 at 3:18 PM, Julia Tylors  wrote:

> to check whether they are equal so that i don't need to make them go
> through another of my operations to save time. That way it will be cached.
>
> On Friday, February 26, 2016 at 12:16:18 PM UTC-8, Stefan Karpinski wrote:
>>
>> Why?
>>
>> On Fri, Feb 26, 2016 at 3:14 PM, Julia Tylors  wrote:
>>
>>> This was a nice question,
>>> i think i am trying to figure out a way to check if 2 functions (partial
>>> possibly) are  at the same syntactic location in the AST and their free
>>> variables refer to the equal/same data
>>>
>>>
>>>
>>> On Friday, February 26, 2016 at 12:02:30 PM UTC-8, Stefan Karpinski
>>> wrote:

 What are you trying to discover about these functions?

 On Fri, Feb 26, 2016 at 2:50 PM, Julia Tylors 
 wrote:

> Isn't there a trick like i can serialize a partial function and then
> check their equality in the serialized form?
>
> On Friday, February 26, 2016 at 11:22:37 AM UTC-8, Stefan Karpinski
> wrote:
>>
>> Functions are compared by identity – they are equal if they are the
>> same function, and not otherwise. Comparing functions syntactically is
>> shallow and nearly useless. Comparing functions by what they compute is
>> undecidable. So identity is essentially the only useful way to compare
>> functions.
>>
>> On Fri, Feb 26, 2016 at 2:09 PM, Julia Tylors 
>> wrote:
>>
>>> Hello,
>>>
>>> I have a simple question. I like to compare functions. For example:
>>>
>>> function some_func(x::Target, y::Config, z::Int64)
>>>#some code here
>>> end
>>>
>>> #some partialization here
>>> f1 = (x::Target,y::Config) -> some_func(x,y,5)
>>> f2 = (x::Target,y::Config) -> some_func(x,y,4)
>>>
>>>
>>> I want to evaluate the following expression:
>>>
>>> f1 == f2
>>>
>>>
>>> Thanks
>>>
>>
>>

>>

[julia-users] make() in Julia for data analysis

[Tom Short]

I'm interested in feedback on the following unregistered package that
implements a build-dependency system like the *make* utility or Ruby's Rake:

https://github.com/tshort/Make.jl  (based on Mike Nolta's Rake.jl)

For me, a typical project will have a script that reads in data followed by
several steps of data processing, analysis, and/or simulation. I normally
split this into several functions. The main problem with this approach is
that I typically have to manually run one or more of these functions to
load variables and do initial processing. Sometimes, I will script this
setup, but it's still klunky.

The package above tries to help manage these dependencies. In theory, I can
include my Julia script and run `make()`. This will load variables and
re-run parts of the data analysis if the input data has changed or I've
change parts of my code.

Suggestions? Other approaches to handle this use case?

Re: [julia-users] Re: Adding another function to Cairo.jl

[Tom Short]

You should be able to test this out locally without modifying Cairo.jl.
Your example code is probably close. You'll need to make sure you qualify
everything that's not exported by Cairo.jl. For example, I think you'll
need to use `Cairo._jl_libcairo` instead of `_jl_libcairo`. If
`CairoContext` isn't exported, you'll need to do the same for that.



On Mon, Jan 25, 2016 at 2:57 PM, Josh Langsfeld  wrote:

> You should test it and decide what to put in it by modifying your own
> forked copy of Cairo.jl. This is what's great about open-source; you get to
> exactly see and control what impact the changes you want will have before
> they get added to the official software.
>
> On Monday, January 25, 2016 at 2:47:05 PM UTC-5, cormu...@mac.com wrote:
>>
>> I'd like to open a pull request, but I wouldn't know exactly what to put
>> in it... :) I was wondering whether I could test the syntax of such a new
>> function first, without actually adding it to the Cairo module. It. It
>> might not be possible, of course...
>
>

Re: [julia-users] CurveFit Pakage: non-linear fitting code

[Tom Short]

One link:

http://stackoverflow.com/questions/34840875/julia-using-curvefit-package-non-linear
On Jan 23, 2016 10:13 AM,  wrote:

> I wish someone would publish a nonlinear fitting code using package
> CurveFit
> using this data, how we can use CurveFit?
>
> x = [0.0 0.2 0.4 1.0 1.6 1.8 2.0 2.6 2.8 3.0 3.8 4.8 5.0 5.2 6.0 6.2 7.4
> 7.6 7.8 8.6 8.8 9.0 9.2 9.4 10.0 10.6 10.8 11.2 11.6 11.8 12.2 12.4];
>
> y = [-0.183 -0.131 0.027 0.3 0.579 0.853 0.935 1.133 1.269 1.102 1.092
> 1.143 0.811 0.91 0.417 0.46 -0.516 -0.334 -0.504 -0.946 -0.916 -0.975
> -1.099 -1.113 -1.297 -1.234 -0.954 -1.122 -0.609 -0.593 -0.403 -0.51];
>

Re: [julia-users] Re: Shell commands in Julia 0.4.2 in Windows 10

[Tom Short]

I use `cd` under shell on Linux, too. It's a nice shortcut.

On Wed, Jan 20, 2016 at 11:20 AM, J Luis  wrote:

> On Windows one can
>
> julia> pwd()
> "C:\\programs\\Julia-0.4"
>
> shell> cd
> C:\j
>
> julia> pwd()
> "C:\\j"
>
> quarta-feira, 20 de Janeiro de 2016 às 15:45:04 UTC, Yichao Yu escreveu:
>>
>> On Wed, Jan 20, 2016 at 10:39 AM, Tony Kelman  wrote:
>> > Julia has portable functions for all of these, you don't need to use
>> shell
>> > mode for cd, dir, readdir, mkdir, rm, mv, or cp.
>>
>> FWIW, you can't use shell for `cd` on any platforms supported by julia.
>>
>> >
>> >
>> >
>> > On Wednesday, January 20, 2016 at 4:56:07 AM UTC-8, Andreas Lobinger
>> wrote:
>> >>
>> >> Hello colleague,
>> >>
>> >> On Wednesday, January 13, 2016 at 11:28:39 PM UTC+1, Tony Kelman
>> wrote:
>> >>>
>> >>> Unix shell commands that we happened to be getting from git are no
>> longer
>> >>> on Julia's path. This is not temporary. Windows is not unix. If you
>> want to
>> >>> use unix shell commands on windows, either download and add your own
>> windows
>> >>> posix layer to your path (temporarily in juliarc, not permanently
>> where it
>> >>> will break other software), or call powershell which has many aliases
>> >>> spelled the same as unix shell commands.
>> >>
>> >>
>> >> i agree, windows is not unix (it never was) and the following might
>> not be
>> >> helpful, but
>> >> Matlab provides (on any OS) a very small set of shell-like commands
>> (cd,
>> >> dir, ls,mkdir, rmdir,  delete, movefile, copyfile).
>> >>
>> >> And providing something like this at a julia REPL should be considered
>> >> user friendly.
>> >>
>> >> Wishing a happy day,
>> >>Andreas
>> >>
>>
>

Re: [julia-users] Steping avg, how faster ?

[Tom Short]

First, don't double post. Second, please read the manual on performance
tips. You've managed to hit most of the beginner problems.

http://docs.julialang.org/en/release-0.4/manual/performance-tips/
On Jan 2, 2016 6:55 AM, "paul analyst"  wrote:

> I have meny lonng vectors (like a)and need avg for intervals (f.e. 100 )
> in second new vecotr (b)
>
> julia> a=rand(10^7);
>
> julia> b=zeros(a);
>
> julia> tic();for i in  [1:100:10^7;]
>b[i:i+99]=mean(a[i:i+99])
>   end;toc()
> elapsed time: 0.138522318 seconds
> 0.138522318
>
> I am looking for more faster way , is any ?
> Paul
>

Re: [julia-users] Why variables passed to this macro are referencing to the module?

[Tom Short]

David Moon, see https://github.com/JuliaLang/julia/pull/10940. It's still
an open WIP.

On Sat, Dec 19, 2015 at 9:51 AM, David Moon  wrote:

> On Saturday, December 19, 2015 at 6:17:18 AM UTC-5, Tim Holy wrote:
>>
>> Presumably you need an esc around the exp. See the metaprogramming
>> chapter.
>>
>
> So the Julia maintainers never pulled my pull request from almost 2 years
> ago that fixed that?
>

Re: [julia-users] DataFrames and special characters

[Tom Short]

This is in the header, right? This has come up a couple of times recently.
There's a new option to readtable to allow nonstandard column names. See:

https://github.com/JuliaStats/DataFrames.jl/pull/896

It's only in the devel version of DataFrames.


On Fri, Dec 18, 2015 at 2:50 PM, Joaquim Masset Lacombe Dias Garcia <
joaquimdgar...@gmail.com> wrote:

> I have noticed that DataFrames´ readtable cannot read by default special
> characters such ac $ % - & and it replaces everything by underscore.
>
> Is there any way to make it read those characters just like the R function
> fread does?
>

Re: [julia-users] optimizing Julia code for numerical integration of PDE

[Tom Short]

Looks like some type instabilities in there. The first hint is the higher
number of allocations. You can dig deeper with this:

@code_warntype ksintegrateUnrolled(u,Lx,dt,Nt);

The variable alpha is not type stable as a start. The inner loop is slow
because of this. You can make the inner loop type stable by putting it in a
separate function you call.

On Thu, Dec 17, 2015 at 4:51 PM, John Gibson  wrote:

> Hi, all. I'm doing a small test case of Julia's speed & code length for
> numerical integration of PDEs, in comparison to Matlab, Python, and C++.
> The test case is the 1d Kuramoto-Sivashinksy equation on a 1d periodic
> domain, using Fourier expansions for spatial discretization and simple
> implicit-explicit finite-difference time-stepping for temporal
> discretization.
>
> My results so far indicate that (for large data sets) a naive Julia code
> runs slightly slower than the line-for-line equivalent Matlab code, faster
> than Python by a factor of two, and slower than an optimized C++ code by a
> factor of two. That's no big deal; the naive code creates lots of temporary
> arrays in the innermost loops.
>
> The trouble is when I try to optimize the Julia by unrolling array
> operations and eliminating the temporary variables, performance is worse.
> Execution is slower and more memory is allocated.
>
> And confusingly, when I do a simplest possible mock-up of the issue with a
> few arrays, the opposite is true. The naive code is slower and allocative,
> the unrolled code is fast and tight.
>
> Here's the mock-up code in file "vecsum.jl" (the operation in the for-
> loop matches the critical operation in the PDE code).
> function vecsumNaive(u,a,b,c,d)
> uf = copy(u);
> for n=1:1000
>   uf = b .* (a .* uf + 1.5*c - 0.5*d)
> end
> sum(uf)
> end
>
>
> function vecsumUnrolled(u,a,b,c,d)
>
> uf = copy(u);
> N = length(a);
> for n=1:1000
> for i=1:N
> @fastmath @inbounds uf[i] = b[i]*(a[i]*uf[i] + 1.5*c[i] -
> 0.5*d[i])
> end
> end
> sum(uf)
> end
>
> N = 512;
> eps = .01;
>
> # Assign random variables of the same types as appear in the KS
> integration code
> a = eps*randn(N);
> b = eps*randn(N);
>
> u = eps*(randn(N)+1im*randn(N));
> c = eps*(randn(N)+1im*randn(N));
> d = eps*(randn(N)+1im*randn(N));
>
> Running this with Julia 0.4.2 on Linux gives
> julia> @time vecsumNaive(u,a,b,c,d);
>   0.433867 seconds (517.62 k allocations: 68.861 MB, 9.93% gc time)
>
> julia> @time vecsumNaive(u,a,b,c,d);
>   0.031010 seconds (60.01 k allocations: 48.684 MB, 18.60% gc time)
>
> julia> @time vecsumNaive(u,a,b,c,d);
>   0.031556 seconds (60.01 k allocations: 48.684 MB, 12.79% gc time)
>
> julia>
>
> julia> @time vecsumUnrolled(u,a,b,c,d);
>   0.025642 seconds (10.56 k allocations: 510.544 KB)
>
> julia> @time vecsumUnrolled(u,a,b,c,d);
>   0.003166 seconds (6 allocations: 8.266 KB)
>
> julia> @time vecsumUnrolled(u,a,b,c,d);
>   0.003533 seconds (6 allocations: 8.266 KB)
>
>
> Great, just as you would expect. The unrolled operation is 10 times faster
> with almost no allocation. But now for the PDE code (file "ksintegrate.jl")
> # The Kuramoto-Shivashinksy eqn is (subscripts indicate differentiation)
> #
> # u_t = -u*u_x - u_xx - u_ on domain [0,Lx] with periodic BCs
> # or
> # u_t = L(u) + N(u)
> #
> # where L(u) = -u_xx - u_ and N(u) = -u u_x
>
> # The numerical integration scheme: Discretize u in space with a Fourier
> expansion
> #
> # u(x,t) = sum_k uf[k] exp(2pi i k x/Lx)
> #
> # Discretize u in time with Crank-Nicolson Adams-Bashforth time stepping
> formula
> # (I - dt/2 L) u^{n+1} = (I + dt/2 L) u^n + 3dt/2 N^n - dt/2 N^{n-1}
> #
> # u^{n+1} = (I - dt/2 L)^{-1} [(I + dt/2 L) u^n + 3dt/2 N^n - dt/2 N^{n-1}]
> #
> # let A = (I + dt/2 L)
> # let B = (I - dt/2 L)^{-1}, then
> #
> # u^{n+1} = B ( A u^n + 3dt/2 N^n - dt/2 N^{n-1})
>
>
> function ksintegrateNaive(u, Lx, dt, Nt);
> # integrate Kuramoto-Shivashinsky eqn for inputs
> #   u  = initial condition, array of gridpoint values of u(x) on uniform
> grid
> #   Lx = periodic domain length
> #   dt = time step
> #   Nt = number of time steps
>
> Nx = length(u)  # number of gridpoints
> kx = vcat(0:Nx/2-1, 0, -Nx/2+1:-1)  # integer wavenumbers:
> exp(2*pi*kx*x/L)
> alpha = 2*pi*kx/Lx  # real wavenumbers:exp(alpha*x)
> D = 1im*alpha;  # spectral D = d/dx operator
> L = alpha.^2 - alpha.^4 # spectral L = -D^2 - D^4 operator
> G = -0.5*D  # spectral -1/2 D operator, to
> eval -u u_x = 1/2 d/dx u^2
>
> # convenience variables
> dt2  = dt/2;
> dt32 = 3*dt/2;
> A =  ones(Nx) + dt2*L;
> B = (ones(Nx) - dt2*L).^(-1);
>
> # evaluate nonlinear term
> Nnf  = G.*fft(u.^2); # Nnf == -1/2 d/dx (u^2) = -u u_x, spectral
> Nn1f = Nnf;  # use Nnf1 = Nnf at first time step
>
> # compute Fourier coeffs of u
> 

Re: [julia-users] Re: Meaty example of using singleton types

[Tom Short]

I'm not sure what you want, either. How about this?

julia> type BadInt{X} end

julia> BadInt{3}()
BadInt{3}()

julia> f{X}(::Type{BadInt{X}}, y) = X - y
f (generic function with 1 method)

julia> f(BadInt{10}, 3)
7

julia> f{X}(::BadInt{X}, y) = X - y
f (generic function with 2 methods)

julia> f(BadInt{10}(), 3)
7


On Wed, Dec 9, 2015 at 8:56 AM, Eric Forgy  wrote:

> Not sure I follow, but does this help?
>
> julia> type BadInt
>end
>
> julia> bi = BadInt()
> BadInt()
>
> julia> typeof(bi)
> BadInt
>
>
> On Wednesday, December 9, 2015 at 9:46:01 PM UTC+8, milktrader wrote:
>>
>> How do you create an instance of type BadInt then?
>>
>> On Wednesday, December 9, 2015 at 7:01:25 AM UTC-5, milktrader wrote:
>>>
>>> Trying to wrap my mind around singleton types to see if they might be
>>> useful for something I'm working on, but running into some confusion. Here
>>> is an example that I started working with:
>>>
>>> julia> type BadInt
>>>end
>>>
>>> julia> import Base.+
>>>
>>> julia> +(x::BadInt, y::Int64) = x - y
>>> + (generic function with 172 methods)
>>>
>>> julia> BadInt() = 2
>>> BadInt
>>>
>>> julia> BadInt + 2
>>> ERROR: MethodError: `+` has no method matching +(::Type{BadInt}, ::Int64)
>>> Closest candidates are:
>>>   +(::Any, ::Any, ::Any, ::Any...)
>>>   +(::Int64, ::Int64)
>>>   +(::Complex{Bool}, ::Real)
>>>   ...
>>>
>>> As I understand, a singleton type can only take on a single value.
>>> What's the utility in supporting this?
>>>
>>

Re: [julia-users] DataFrames : Apply a function by rows

[Tom Short]

Contributions/pull requests from folks that need that are welcome. I don't
have that need. For row operations, I can generally get by with loops or
`@byrow!` in DataFramesMeta.
On Nov 22, 2015 8:23 AM, "Fred"  wrote:

> Yes, it is a good solution, but it means that DataFrames cannot be used to
> do some calculations by rows, it is a severe limitation. An equivalent of 
> colwise()
> whould be very usefull.
>
> Le dimanche 22 novembre 2015 14:11:21 UTC+1, tshort a écrit :
>>
>> I'd convert the whole DataFrame to a matrix and use a loop over rows.
>>
>

Re: [julia-users] DataFrames : Apply a function by rows

[Tom Short]

I'd convert the whole DataFrame to a matrix and use a loop over rows.
On Nov 22, 2015 2:54 AM, "Fred"  wrote:

> In my last example, the function mean() is not well chosen. In fact, what
> I would like to calculate is a statistical test line by lline, like TTest,
> or Wilcoxon. This is why I need to iterate thought 2 DataFrames at the same
> time if I subset the DataFrame first to increase speed :)
>
>
> Something like :
>
> julia> for r1,r2 in eachrow(df1, df2)
>   println(TTest(r1,r2))
>end
> ERROR: syntax: invalid iteration specification
>
>
>
>
> Le samedi 21 novembre 2015 19:17:27 UTC+1, Fred a écrit :
>>
>> It is a good idea but how is it possible to iterate two dataframes at the
>> same time ? Something like :
>>
>> julia> df = DataFrame(a=1:5, b=7:11, c=10:14, d=20:24)
>> 5x4 DataFrames.DataFrame
>> | Row | a | b  | c  | d  |
>> |-|---||||
>> | 1   | 1 | 7  | 10 | 20 |
>> | 2   | 2 | 8  | 11 | 21 |
>> | 3   | 3 | 9  | 12 | 22 |
>> | 4   | 4 | 10 | 13 | 23 |
>> | 5   | 5 | 11 | 14 | 24 |
>>
>> julia> df1 = df[1:2,]
>> 5x2 DataFrames.DataFrame
>> | Row | a | b  |
>> |-|---||
>> | 1   | 1 | 7  |
>> | 2   | 2 | 8  |
>> | 3   | 3 | 9  |
>> | 4   | 4 | 10 |
>> | 5   | 5 | 11 |
>>
>> julia> df1 = df[3:4,]
>> 5x2 DataFrames.DataFrame
>> | Row | c  | d  |
>> |-|||
>> | 1   | 10 | 20 |
>> | 2   | 11 | 21 |
>> | 3   | 12 | 22 |
>> | 4   | 13 | 23 |
>> | 5   | 14 | 24 |
>>
>> julia> for r1,r2 in eachrow(df1, df2)
>>   println(mean(r1,r2))
>>end
>> ERROR: syntax: invalid iteration specification
>>
>>
>>
>>
>> Le samedi 21 novembre 2015 15:08:34 UTC+1, tshort a écrit :
>>>
>>> For the subset, do the indexing after the conversion to an array, or
>>> subset the DataFrame first (probably faster).
>>>
>>

Re: [julia-users] DataFrames : Apply a function by rows

[Tom Short]

You can try `eachrow`. It probably won't be fast, though. Here's an example:

https://github.com/JuliaStats/DataFrames.jl/blob/master/test/iteration.jl#L34
On Nov 21, 2015 7:19 AM, "Fred"  wrote:

> Hi,
>
> In DataFrames, it is easy to apply a function by columns using the
> colwise() function. But I find very difficult and inefficient to apply a
> function by rows.
>
> For example :
>
>
>
>  julia> df = DataFrame(a=1:5, b=7:11, c=10:14)
> 5x3 DataFrames.DataFrame
> | Row | a | b  | c  |
> |-|---|||
> | 1   | 1 | 7  | 10 |
> | 2   | 2 | 8  | 11 |
> | 3   | 3 | 9  | 12 |
> | 4   | 4 | 10 | 13 |
> | 5   | 5 | 11 | 14 |
>
>
>
>  julia> colwise(mean,df)
> 3-element Array{Any,1}:
>  [3.0]
>  [9.0]
>  [12.0]
>
>
>  julia> colwise(mean,df[1,1:2])
> 2-element Array{Any,1}:
>  [1.0]
>  [7.0]
>
>
>
> To calculate the mean of a row (or a subset), the only way I found is this
> :
>
> julia> mean(convert(Array,df[1,1:3]))
> 6.0
>
>
>
> I think this is inefficient and probably very slow. I there a better way
> to apply a function by rows ?
>
> Thanks !
>

Re: [julia-users] DataFrames : Apply a function by rows

[Tom Short]

For the subset, do the indexing after the conversion to an array, or subset
the DataFrame first (probably faster).
On Nov 21, 2015 8:43 AM, "Fred"  wrote:

> Thanks for the answer. I tried "eachrow" but I have 2 problems :
>
> 1- I still have to do an array conversion, I think it is slow
>
> julia> for r in eachrow(df)
>   println(mean(convert(Array,r)))
>end
> 6.0
> 7.0
> 8.0
> 9.0
> 10.0
>
>
>
> 2- I do not manage to use a subset of the row, for example the 2 first
> values :
> julia> for r in eachrow(df)
>   println(mean(convert(Array,r)))
>end
> 6.0
> 7.0
> 8.0
> 9.0
> 10.0
>
> julia> for r in eachrow(df)
>   println(mean(convert(Array,r[1:2])))
>end
> WARNING: [a] concatenation is deprecated; use collect(a) instead
>  in depwarn at deprecated.jl:73
>  in oldstyle_vcat_warning at ./abstractarray.jl:29
>  [inlined code] from none:2
>  in anonymous at no file:0
> while loading no file, in expression starting on line 0
> 4.0
>
>
>
>
>
>
> Le samedi 21 novembre 2015 14:04:11 UTC+1, tshort a écrit :
>>
>> You can try `eachrow`. It probably won't be fast, though. Here's an
>> example:
>>
>>
>> https://github.com/JuliaStats/DataFrames.jl/blob/master/test/iteration.jl#L34
>> On Nov 21, 2015 7:19 AM, "Fred"  wrote:
>>
>>> Hi,
>>>
>>> In DataFrames, it is easy to apply a function by columns using the
>>> colwise() function. But I find very difficult and inefficient to apply
>>> a function by rows.
>>>
>>> For example :
>>>
>>>
>>>
>>>  julia> df = DataFrame(a=1:5, b=7:11, c=10:14)
>>> 5x3 DataFrames.DataFrame
>>> | Row | a | b  | c  |
>>> |-|---|||
>>> | 1   | 1 | 7  | 10 |
>>> | 2   | 2 | 8  | 11 |
>>> | 3   | 3 | 9  | 12 |
>>> | 4   | 4 | 10 | 13 |
>>> | 5   | 5 | 11 | 14 |
>>>
>>>
>>>
>>>  julia> colwise(mean,df)
>>> 3-element Array{Any,1}:
>>>  [3.0]
>>>  [9.0]
>>>  [12.0]
>>>
>>>
>>>  julia> colwise(mean,df[1,1:2])
>>> 2-element Array{Any,1}:
>>>  [1.0]
>>>  [7.0]
>>>
>>>
>>>
>>> To calculate the mean of a row (or a subset), the only way I found is
>>> this :
>>>
>>> julia> mean(convert(Array,df[1,1:3]))
>>> 6.0
>>>
>>>
>>>
>>> I think this is inefficient and probably very slow. I there a better way
>>> to apply a function by rows ?
>>>
>>> Thanks !
>>>
>>

Re: [julia-users] ode-sundials-events option as in deSolve (R)

[Tom Short]

It's hard to tell from that. Rather than clog this mailing list with
debugging emails, please file an issue at the Sims repo. Provide your Julia
version in that, too.

On Mon, Aug 24, 2015 at 9:27 AM, Martin J jmalph...@gmail.com wrote:

 Thanks for your reply.

 I used your code with some changes.
 
 using Sims, Winston

 function model()
 # note that the time scale is in hours
 D = Unknown(10.0, D)
 A1 = Unknown(A1)
 rateConstant1 = 3.0
 rateConstant2 = 5.0
 Equation[
 der(D) = -rateConstant1*D
 der(A1) = rateConstant1*D - rateConstant2*A1
 Event(sin(2pi/12 * MTime), # Initiate an event every 12 hours
   Equation[
   reinit(D, D + 10)  # add a dosage of 10
   ])
 ]
 end
 x = sim(model(), 60);
 wplot(x)

 *

 I get the following error

 sim not defined


 I understand that Julia does not realise the function sim()


 Any suggestions ?


 regards,

 Martin


 On Friday, August 21, 2015 at 1:41:09 PM UTC+1, tshort wrote:

 One way is with the Sims package. I'm not sure I got everything right
 with your model.

 https://tshort.github.io/Sims.jl/

 using Sims, Winston

 function model()
 # note that the time scale is in hours
 D = Unknown(10.0, D)
 A1 = Unknown(A1)
 rateConstant1 = 3.0
 rateConstant2 = 5.0
 @equations begin
 der(D) = -rateConstant1*D
 der(A1) = rateConstant1*D - rateConstant2*A1
 Event(sin(2pi/12 * MTime), # Initiate an event every 12 hours
   Equation[
   reinit(D, D + 10)  # add a dosage of 10
   ])
 end
 end

 x = sim(model(), 60);
 wplot(x)


 On Thu, Aug 20, 2015 at 8:36 PM, Martin J jmal...@gmail.com wrote:

 Hi all,

 I am trying to use julia ode or sundials for the following simple
 differential equation.

 dD = -rateConstant1*D
 dA1 = rateConstant1*D - rateConstant2*A1

 This differential equations explain the disposition of drug in human
 system.

 *My question is related to resetting state variable at specified time. *

 In this example, drug was administered repeatedly every 12 hours for 48
 hours. So, I need to add an amount to dD every 12 hour to achieve this. In
 R and deSolve package, this repeated addition of dose was achieved by using
 ‘events’ options in ode. (R code attached)

 dose.events (please refer R code - line 32) object is presented below:

 var time value   method
 depot0 10 replace
 depot   12 10 replace
 depot   24 10 replace
 depot   36 10 replace
 depot   48 10 replace

 With my limited understanding, this ‘events’ option in ode (deSolve),
 replaces the depot (dD) at these mentioned time with the said value.

 I am looking for similar option in sundials or julia ode.

 Could you please guide me with this ?

 Thanks for your suggestions.

 regards,
 Martin

Re: [julia-users] Slicing DataFrames resets indexes

[Tom Short]

DataFrames don't have real row indexes. If you want a row identifier, you
can add it as an extra column.
On Aug 23, 2015 4:32 PM, Robert Smith rsmith31...@gmail.com wrote:

 Very silly question. In Julia 0.3, I noticed that after slicing some
 DataFrame, the resulting object doesn't keep its original indexes. For
 example:

 julia using DataFrames

 julia df = DataFrame(x=1:100)

 julia df[20:50,:]
 31x1 DataFrame
 | Row | x  |
 |-||
 | 1   | 20 |
 | 2   | 21 |
 | 3   | 22 |
 | 4   | 23 |
 | 5   | 24 |
 | 6   | 25 |
 | 7   | 26 |
 | 8   | 27 |
 ⋮
 | 23  | 42 |
 | 24  | 43 |
 | 25  | 44 |
 | 26  | 45 |
 | 27  | 46 |
 | 28  | 47 |
 | 29  | 48 |
 | 30  | 49 |
 | 31  | 50 |

 In the previous line, the DataFrame's indexes start with 1 instead of 20.
 Is there a way to keep the same index as the original DataFrame (this is
 the default behavior in Pandas and R).

 Thanks.

Re: [julia-users] ode-sundials-events option as in deSolve (R)

[Tom Short]

One way is with the Sims package. I'm not sure I got everything right with
your model.

https://tshort.github.io/Sims.jl/

using Sims, Winston

function model()
# note that the time scale is in hours
D = Unknown(10.0, D)
A1 = Unknown(A1)
rateConstant1 = 3.0
rateConstant2 = 5.0
@equations begin
der(D) = -rateConstant1*D
der(A1) = rateConstant1*D - rateConstant2*A1
Event(sin(2pi/12 * MTime), # Initiate an event every 12 hours
  Equation[
  reinit(D, D + 10)  # add a dosage of 10
  ])
end
end

x = sim(model(), 60);
wplot(x)


On Thu, Aug 20, 2015 at 8:36 PM, Martin J jmalph...@gmail.com wrote:

 Hi all,

 I am trying to use julia ode or sundials for the following simple
 differential equation.

 dD = -rateConstant1*D
 dA1 = rateConstant1*D - rateConstant2*A1

 This differential equations explain the disposition of drug in human
 system.

 *My question is related to resetting state variable at specified time. *

 In this example, drug was administered repeatedly every 12 hours for 48
 hours. So, I need to add an amount to dD every 12 hour to achieve this. In
 R and deSolve package, this repeated addition of dose was achieved by using
 ‘events’ options in ode. (R code attached)

 dose.events (please refer R code - line 32) object is presented below:

 var time value   method
 depot0 10 replace
 depot   12 10 replace
 depot   24 10 replace
 depot   36 10 replace
 depot   48 10 replace

 With my limited understanding, this ‘events’ option in ode (deSolve),
 replaces the depot (dD) at these mentioned time with the said value.

 I am looking for similar option in sundials or julia ode.

 Could you please guide me with this ?

 Thanks for your suggestions.

 regards,
 Martin

Re: [julia-users] Multidimensional DataArray as column of DataFrame

[Tom Short]

DataFrames was written with the intent that columns are vector-like
objects. Sometimes other objects can be used, but support of features gets
iffy. Options that might work include:

* Wrap your multidimensional object in a vector type that returns the
appropriate subset when indexed by a row. That will work with the existing
DataFrames.

* Submit a pull request to DataFrames to better support multidimensional
objects.

On Fri, May 22, 2015 at 7:39 AM, Josef Sachs sa...@cyb.org wrote:

 DataFrames apparently do not allow columns to be multidimensional
 DataArrays.

 julia DataFrame(A = 1:5, B = DataArray(reshape(1:15,(5,3
 ERROR: ArgumentError: setindex!(::DataFrame, ...) only broadcasts scalars,
 not arrays
  in setindex! at
 C:\Users\s2sajs\.julia\v0.4\DataFrames\src\dataframe\dataframe.jl:356

 Is that a bug, or an unintentional restriction, or an intentional
 restriction?

Re: [julia-users] writing a DataFrame consisting of DataFrames to a table

[Tom Short]

A  dataframe isn't really meant to be used as a column in another
DataFrame. It may work for a few things, but it's likely to be broken for
others.
On May 8, 2015 4:56 AM, axsk smartmin...@gmail.com wrote:

 I need to write and then read a DataFrame which consists of other
 DataFrames to a file.

 df=DataFrame(d=DataFrame(x=[1:5]))
 gives the desired DataFrame, but
 writetable(df.csv,df)
 readtable(df.csv)
 returns:
 1x1 DataFrames.DataFrame
 | Row | d
   |

 |-|-|
 | 1   | 5x1 DataFrames.DataFrame\n| Row | x |\n|-|---|\n| 1   | 1
 |\n| 2   | 2 |\n| 3   | 3 |\n| 4   | 4 |\n| 5   | 5 | |

 Thus the interiour DataFrame gets stored as string in the file.
 Is there anyway to properly read the inner DataFrame?

Re: [julia-users] Panel Data using DataFrame (or other method?)

[Tom Short]

Yes, that'd be a good place to start.

On Thu, May 7, 2015 at 7:13 AM, Nils Gudat nils.gu...@gmail.com wrote:

 Two problems with this:

 (i) I'm a terrible, terrible programmer (just as Mike about the time I
 wanted to help out by making his Jewel package 0.4-compatible)
 (ii) The project I'm working on right now has a tight deadline, so I won't
 have much time for the exploratory stage and will in all likelihood have
 to revert to stuff that is working now (i.e. pandas or plm)

 On the other hand, it would probably be a great thing in terms of learning
 something about Julia and programming in general, so if I can make time I
 should try it. So your suggestion would be to start from dataframe.jl
 https://github.com/JuliaStats/DataFrames.jl/blob/master/src/dataframe/dataframe.jl
 and then try to write something similar, defining a type PanelDataFrame
 which extends the functionality of the standard DataFrame?

Re: [julia-users] Panel Data using DataFrame (or other method?)

[Tom Short]

The AxisArrays package may be useful, depending on your data. The data must
all be the same type. This package is still young and requires the
development version of Julia.

https://github.com/mbauman/AxisArrays.jl


On Tue, May 5, 2015 at 9:35 AM, Kevin Squire kevin.squ...@gmail.com wrote:

 Hi Nils,

 Unfortunately, this does not exist in Julia.  You could do an array of
 DataFrames, but you won't be able to do most of the things Panels allow.
 If you want to stay in Julia, you might check out Pandas.jl, which calls
 out to Python Pandas.

 Cheers,
Kevin

 On Tue, May 5, 2015 at 3:05 AM, Nils Gudat nils.gu...@gmail.com wrote:

 Is it possible to construct a panel dataset using a DataFrame or any
 other method in Julia?
 Pandas offers this (pandas.Panel
 http://pandas.pydata.org/pandas-docs/stable/dsintro.html#panel),
 although the docs are saying this is a less used data structure. What is
 the closest I can get in Julia?

Re: [julia-users] Panel Data using DataFrame (or other method?)

[Tom Short]

I second Jacob's suggestion. It's surprising how fast you can draft out a
new type and get useful functionality. And, you can use code from
DataFrames or other packages as a starting point.

On Tue, May 5, 2015 at 12:56 PM, Jacob Quinn quinn.jac...@gmail.com wrote:

 Remember, the world is your oyster! Take a stab at creating a package and
 sharing it! I'm sure you'd get some interest/feedback.

 -Jacob

 On Tue, May 5, 2015 at 10:53 AM, Nils Gudat nils.gu...@gmail.com wrote:

 Hm, that's a shame - I was hoping for something better than pandas' panel
 implementation in Julia, as I'm not a big fan of it (nor of R's plm
 package, but I guess that's what I'll have to revert to now).

Re: [julia-users] Holt-Winters D-E Smoothing in Julia

[Tom Short]

Given the lack of replies, I'd guess that the answer is no. Searching
holt-winters julia jl didn't turn up anything.

Your options might be:

* Use RCall to call the function in R. RCall is still rather young.

* Use PyCall to call a function in Python. PyCall is fairly mature.

* Implement your own version in Julia. As a starting point, you can use R
or Python code. The following Python code is more compatible with the MIT
license used by StatsBase.jl than the R code:

https://pythonhosted.org/pycast/_modules/pycast/methods/exponentialsmoothing.html#ExponentialSmoothing

The best starting point might be the following MIT-licensed code in Ruby
that looks pretty clean.

https://github.com/cmdrkeene/holt_winters/blob/master/lib/holt_winters.rb



On Tue, Apr 7, 2015 at 5:48 PM, Philip Tellis philip.tel...@gmail.com
wrote:

 Does anyone know if there's a Holt-Winters Double-Exponential Smoothing
 module for Julia?

 It's available in R:
 http://astrostatistics.psu.edu/su07/R/html/stats/html/HoltWinters.html

Re: [julia-users] Re: Confidence for regression

[Tom Short]

I think the question was for prediction intervals. I don't see that in GLM,
yet.

On Wed, Apr 1, 2015 at 12:48 PM, verylucky...@gmail.com wrote:

 GLM.jl have prediction methods for Linear and Generalized Linear Models.
 They take corresponding models and features as input. Please see for
 implementation details
 [glm]
 https://github.com/JuliaStats/GLM.jl/blob/a7fb0057a7bc835d819e842c6f42f14601840a1b/src/glmfit.jl#L249
 and
 [lm]
 https://github.com/JuliaStats/GLM.jl/blob/4f862cf11a93d2c91ce5e745f51233c49941f836/src/lm.jl#L77

 If your inputs are in dataframe format, DataFrames.jl gives nice wrappers
 so that you can input DataFrameRegressionModel and DataFrames instead.
 Please see for details

 https://github.com/JuliaStats/DataFrames.jl/blob/b6e65259c9b0d74187f06cb6e4e9302b9f1c9106/src/statsmodels/statsmodel.jl#L69

 These implementations are overloads of functions in StatsBase
 https://github.com/JuliaStats/StatsBase.jl/blob/e6411b644c02d470036d44fe25a49cdf89a15fff/src/statmodels.jl#L21

 I have used these methods with success. Please let me know if these don't
 work for your case.

 Best wishes!


 On Wednesday, April 1, 2015 at 1:50:08 AM UTC-4, Christopher Fusting wrote:

 Thanks. I've found confidence intervals, still looking for prediction.

 Cheers,

 _Chris

 On Tuesday, March 31, 2015 at 4:53:37 PM UTC-4, Patrick Kofod Mogensen
 wrote:

 I do believe GLM.jl has this.

Re: [julia-users] Extending a DataFrame (or, why aren't my imports working?)

[Tom Short]

Consider putting the code somewhere, so we can take a look.

On Fri, Mar 27, 2015 at 1:55 PM, kevin.dale.sm...@gmail.com wrote:

 I did try that as well, but it acted the same way.  The size function is
 actually defined in abstractdataframe.jl as Base.size, and that's where the
 error message originates from.  I don't know if that has anything to do
 with it though.

Re: [julia-users] Extending a DataFrame (or, why aren't my imports working?)

[Tom Short]

When you define your version of nrow, are you extending the DataFrames
version as in the following?

DataFrames.nrow(df::MyDataFrame) = ...

If not, you are defining your own nrow that is different than the one in
DataFrames.



On Fri, Mar 27, 2015 at 12:45 PM, kevin.dale.sm...@gmail.com wrote:

 I'm trying to extend DataFrames so that I can include metadata on the
 dataframe and the columns.  Unfortunately, from the way I understand how
 Julia works, this is not an easy task. It seams as though I pretty much
 have to copy the existing dataframes.jl file and replace all of the
 DataFrame references with MyDataFrame, then add in the metadata parts
 where needed (or use composition and proxy all of the DataFrame
 interfaces).  This method seems to work for some things (it works with
 Gadfly), however, I can't seem to get ncol and nrow to work properly.  When
 I try to do size(mydf), I get the following error:

 `nrow` has no method matching nrow(::MyDataFrame)

 However, if I do methods(nrow), it displays this:

nrow(df::MyDataFrame)

 Which is exactly what the previous message said didn't exist.  I'm a
 little puzzled as to why DataFrames' original nrow doesn't show up in that
 output as well since it is exported.  When I do a using DataFrames in
 this same session.  I get the following warning.

Warning: using DataFrames.nrow in module Main conflicts with an
 existing identifier.

 I think it's safe to say that I'm pretty confused at this point.  I'd
 appreciate it if someone could clarify how extending existing structures is
 supposed to work.

Re: [julia-users] Extending a DataFrame (or, why aren't my imports working?)

[Tom Short]

You need to import index from DataFrames. That's one reason I prefer using
`using DataFrames` and defining methods with `DataFrames.index() = ...`.

On Fri, Mar 27, 2015 at 2:46 PM, kevin.dale.sm...@gmail.com wrote:

 Ok, I narrowed it down to a very small test case.  The mymodule.jl file is
 at the bottom of this posting.  If you save that to a file then run this
 code, you'll get the same effect as my original problem except with the
 'index' method.

 using mymodule

 mydf = MyDataFrame(Any[], Index())

 # This line complains that `index` has no method matching
 index(::MyDataFrame)
 display(mydf)

 # This displays my `index` method
 methods(index)

 # This shows that my `index` method works
 index(mydf)


 === mymodule.jl ===

 module mymodule

 import DataFrames: AbstractDataFrame, DataFrame, Index, nrow, ncol
 import DataArrays: DataArray

 export MyDataFrame, nrow, ncol, Index, index, columns

 type MyDataFrame : AbstractDataFrame
columns::Vector{Any}
colindex::Index

function MyDataFrame(columns::Vector{Any}, colindex::Index)

   ncols = length(columns)
   if ncols  1
   nrows = length(columns[1])
   equallengths = true
   for i in 2:ncols
   equallengths = length(columns[i]) == nrows
   end
   if !equallengths
   msg = All columns in a DataFrame must be the same length
   throw(ArgumentError(msg))
   end
   end

   if length(colindex) != ncols
   msg = Columns and column index must be the same length
   throw(ArgumentError(msg))
   end

   new(columns, colindex)
end

 end

 index(df::MyDataFrame) = df.colindex
 columns(df::MyDataFrame) = df.columns

 nrow(df::MyDataFrame) = ncol(df)  0 ? length(df.columns[1])::Int : 0
 ncol(df::MyDataFrame) = length(index(df))

 end

 ==

Re: [julia-users] Working with a custom type - questions

[Tom Short]

Can you just use the DateTime type in Julia?

http://julia.readthedocs.org/en/latest/stdlib/dates/

This is in the development version of Julia. For a backwards-compatible
implementation for Julia 0.3, see:

https://github.com/quinnj/Dates.jl

If the DateTime type isn't suitable (for example, millisecond resolution),
you could possibly inherit from suitable types and still get more features
than you currently have.



On Mon, Mar 16, 2015 at 4:00 PM, Chris 7hunderstr...@gmail.com wrote:

 Hello,

 I have written a bunch of Julia code with functions assuming a certain
 variable (time) is a Float64 (really it's a Julian Date). I recently
 decided it might be a good idea to introduce a custom JDate type, which I
 defined as a subtype of FloatingPoint:

 immutable JDate : FloatingPoint

  t::Float64

 end


  My thinking being that this would help avoid ambiguity with my code, as I
 sometimes work with other units of time (e.g. seconds) that are also
 floating point numbers. Of course, this introduced a number of problems in
 my existing code, including

1. Arithmetic operators not defined for type JDate. My workaround for
this has thus far been to overload the operators to accept JDate types.
2. My functions not having methods that accept time as a JDate type.
My workaround for this has been to change the function to accept 
 time::FloatingPoint
instead of time::Float64, but this workaround does not work for Arrays.

 I've read through the relevant parts of the documentation, although I'm
 not sure how much actually stuck, since a good portion was over my head.

 In general, am I going about this the right way? If not, what should I do
 differently? Is there any way to resolve the issue with Arrays, besides
 creating another method that accepts Array{JDate,1}?

 Thanks in advance,
 Chris

Re: [julia-users] Automatic doc tools for Julia

[Tom Short]

Here is an example of documentation for a package I maintain:

https://tshort.github.io/Sims.jl/

Here are examples of docstrings:

https://github.com/tshort/Sims.jl/blob/master/src/sim.jl#L1-L96

Here is the config file for Mkdocs:

https://github.com/tshort/Sims.jl/blob/master/mkdocs.yml

Here is a Julia script that uses the Lexicon package to build the API
documentation from the docstrings:

https://github.com/tshort/Sims.jl/blob/master/docs/build.jl

Here are other packages that use Mkdocs:

https://www.google.com/search?q=mkdocs.yml+jl+site:github.comie=utf-8oe=utf-8


On Thu, Mar 12, 2015 at 10:28 AM, Ismael VC ismael.vc1...@gmail.com wrote:

 tshort, could you provide us an example please?

 El jueves, 12 de marzo de 2015, 4:59:14 (UTC-6), tshort escribió:

 The Lexicon package works well for me along with Mkdocs.
 On Mar 12, 2015 6:03 AM, Ján Adamčák jada...@gmail.com wrote:

 Hi guys,

 Can I ask you for something like best practice with auto doc tools for
 parsing Julia code? I try use Doxygen and Sphinx, but I think this is not
 good solutions in this timeversion(0.3.6). And/Or some tool for generate
 UML diagrams from julia code?

 Thanks.

 P.S.:
 My idea with this thread is generate something like big manual of
 knowlege how to use auto doc tools in Julia.

Re: [julia-users] Automatic doc tools for Julia

[Tom Short]

The Lexicon package works well for me along with Mkdocs.
On Mar 12, 2015 6:03 AM, Ján Adamčák jadam...@gmail.com wrote:

 Hi guys,

 Can I ask you for something like best practice with auto doc tools for
 parsing Julia code? I try use Doxygen and Sphinx, but I think this is not
 good solutions in this timeversion(0.3.6). And/Or some tool for generate
 UML diagrams from julia code?

 Thanks.

 P.S.:
 My idea with this thread is generate something like big manual of knowlege
 how to use auto doc tools in Julia.

Re: [julia-users] DataFramesMeta

[Tom Short]

What platform?

It'd help to file an issue.
On Mar 8, 2015 5:41 PM, pip7k...@gmail.com wrote:

 Hi
 Using version Julia 0.3.6 and trying to run  ..
 using DataFramesMeta .. it just hangs!
 Any ideas.

 Regards

Re: [julia-users] Basic DataFrame Subsetting Question

[Tom Short]

If you must use eval, putting the expression inside a function can help
work around issues. I didn't try it, but it'd look something like:



using DataFrames

df = DataFrame(x1 = 1:3, x2 = [2, 1, 2],x3 = [22, 21, 20]);
ex=:(myfun(df) = df[(df[:x3] . 22)  (df[:x2] .== 2), :])
eval(ex)
df_filtered=myfun(df)



On Tue, Mar 3, 2015 at 2:28 AM, bernhard kafis...@gmail.com wrote:

 Thank you Tom. This works fine.
 But what if I have an expression? Let's say I derive the expression ex by
 some algorithm (e.g. a decision tree) and then want to apply it to a
 DataFrame. How do I get the last line of this code to work?


 using DataFrames
 using DataFramesMeta


 df = DataFrame(x1 = 1:3, x2 = [2, 1, 2],x3 = [22, 21, 20]);
 df[(df[:x3] . 22)  (df[:x2] .== 2), :]
 df_filtered=@where(df, (:x3 . 22)  (:x2 .== 2))
 ex=:((:x3 . 22)  (:x2 .== 2))
 df_filtered2=@where(df,eval(ex))




 Am Montag, 2. März 2015 17:29:30 UTC+1 schrieb tshort:

 Here are a couple of ways:

 julia df = DataFrame(x1 = 1:3, x2 = [2, 1, 2],x3 = [22, 21, 20]);

 julia df[(df[:x3] . 22)  (df[:x2] .== 2), :]
 1x3 DataFrames.DataFrame
 | Row | x1 | x2 | x3 |
 |-||||
 | 1   | 3  | 2  | 20 |

 julia using DataFramesMeta

 julia @where(df, (:x3 . 22)  (:x2 .== 2))
 1x3 DataFrames.DataFrame
 | Row | x1 | x2 | x3 |
 |-||||
 | 1   | 3  | 2  | 20 |


 On Mon, Mar 2, 2015 at 3:09 AM, bernhard kafi...@gmail.com wrote:

 Hi all

 Can anyone tell me how to make this work in the current version of
 Julia? v0.3.4
 I would even prefer to be able to evaluate an expression and receive the
 index of the rows which match the results. The code could look similar to
 this

 df = DataFrame(x1 = 1:3, x2 = [2, 1, 2],x3 = [22, 21, 20])
 ex=:((x3 . 22)  (x2 .== 2))

 rows_matching_the_condition=df[eval(ex)] #if this would work, it would
 return the subset I guess and not the index vector though


 Am Freitag, 27. September 2013 22:38:49 UTC+2 schrieb Stefan Karpinski:

 Using single  and | should work too.

 On Friday, September 27, 2013, Andrew Gendreau wrote:

 Makes sense, thanks Joosep

 On Friday, September 27, 2013 4:04:13 PM UTC-4, Joosep Pata wrote:

 The following works as one would expect.

 df[:((a .== 1) .+ (b .== 2)), :] #or
 df[:((a .== 1) .* (b .== 2)), :] #and

 My non-expert explanation is the following:
 both subexpressions return a DataArray of bools, for which the
 logical operator  is not defined (what would it mean for an array to 
 have
 a single true/false value?), but the elementwise product via .* is. The
 DataFrame is masked via the resulting vector of bools.

 Joosep

 On Sep 27, 2013, at 10:49 PM, Andrew Gendreau agen...@gmail.com
 wrote:

  Hi,
 
  I'm just getting started with working with programming languages in
 general and am trying to learn Julia alongside R.  I'm running into a
 problem when subsetting within DataFrames in Julia, specifically when I 
 try
 to do so based on two conditions.  For reference, I'm working off the
 syntax on the DataFrames subsetting page:
 http://juliastats.github.io/DataFrames.jl/subsets.html
 
  These work as expected:
 
  julia df2[:(Ozone . 5), :]
  5x2 DataFrame:
  Ozone Temp
  [1,]6   86
  [2,]7   87
  [3,]8   88
  [4,]9   89
  [5,]   10   90
 
 
  julia df2[:(Temp . 88), :]
  2x2 DataFrame:
  Ozone Temp
  [1,]9   89
  [2,]   10   90
 
 
  But when I try to combine them:
 
 
  julia df2[:(Ozone . 5  Temp . 88), :]
  ERROR: type: non-boolean (DataArray{Bool,1}) used in boolean
 context
   in anonymous at C:\Users\andrew.gendreau\AppData\Roaming\Julia\
 packages\DataFrames\src\dataframe.jl:1466
 
 
  julia df2[:(Ozone . 5  Temp . 88), :]
  0x2 DataFrame:
Ozone Temp
 
 
  Any ideas on what I'm doing wrong?   select(:(Ozone . 5  Temp .
 88), df2)  doesn't seem to work either (returns an empty SubDataFrame).
 
  Thank you,
  Andrew

Re: [julia-users] Basic DataFrame Subsetting Question

[Tom Short]

Here are a couple of ways:

julia df = DataFrame(x1 = 1:3, x2 = [2, 1, 2],x3 = [22, 21, 20]);

julia df[(df[:x3] . 22)  (df[:x2] .== 2), :]
1x3 DataFrames.DataFrame
| Row | x1 | x2 | x3 |
|-||||
| 1   | 3  | 2  | 20 |

julia using DataFramesMeta

julia @where(df, (:x3 . 22)  (:x2 .== 2))
1x3 DataFrames.DataFrame
| Row | x1 | x2 | x3 |
|-||||
| 1   | 3  | 2  | 20 |


On Mon, Mar 2, 2015 at 3:09 AM, bernhard kafis...@gmail.com wrote:

 Hi all

 Can anyone tell me how to make this work in the current version of Julia?
 v0.3.4
 I would even prefer to be able to evaluate an expression and receive the
 index of the rows which match the results. The code could look similar to
 this

 df = DataFrame(x1 = 1:3, x2 = [2, 1, 2],x3 = [22, 21, 20])
 ex=:((x3 . 22)  (x2 .== 2))

 rows_matching_the_condition=df[eval(ex)] #if this would work, it would
 return the subset I guess and not the index vector though


 Am Freitag, 27. September 2013 22:38:49 UTC+2 schrieb Stefan Karpinski:

 Using single  and | should work too.

 On Friday, September 27, 2013, Andrew Gendreau wrote:

 Makes sense, thanks Joosep

 On Friday, September 27, 2013 4:04:13 PM UTC-4, Joosep Pata wrote:

 The following works as one would expect.

 df[:((a .== 1) .+ (b .== 2)), :] #or
 df[:((a .== 1) .* (b .== 2)), :] #and

 My non-expert explanation is the following:
 both subexpressions return a DataArray of bools, for which the logical
 operator  is not defined (what would it mean for an array to have a
 single true/false value?), but the elementwise product via .* is. The
 DataFrame is masked via the resulting vector of bools.

 Joosep

 On Sep 27, 2013, at 10:49 PM, Andrew Gendreau agen...@gmail.com
 wrote:

  Hi,
 
  I'm just getting started with working with programming languages in
 general and am trying to learn Julia alongside R.  I'm running into a
 problem when subsetting within DataFrames in Julia, specifically when I try
 to do so based on two conditions.  For reference, I'm working off the
 syntax on the DataFrames subsetting page:  http://juliastats.github.io/
 DataFrames.jl/subsets.html
 
  These work as expected:
 
  julia df2[:(Ozone . 5), :]
  5x2 DataFrame:
  Ozone Temp
  [1,]6   86
  [2,]7   87
  [3,]8   88
  [4,]9   89
  [5,]   10   90
 
 
  julia df2[:(Temp . 88), :]
  2x2 DataFrame:
  Ozone Temp
  [1,]9   89
  [2,]   10   90
 
 
  But when I try to combine them:
 
 
  julia df2[:(Ozone . 5  Temp . 88), :]
  ERROR: type: non-boolean (DataArray{Bool,1}) used in boolean context
   in anonymous at C:\Users\andrew.gendreau\AppData\Roaming\Julia\
 packages\DataFrames\src\dataframe.jl:1466
 
 
  julia df2[:(Ozone . 5  Temp . 88), :]
  0x2 DataFrame:
Ozone Temp
 
 
  Any ideas on what I'm doing wrong?   select(:(Ozone . 5  Temp .
 88), df2)  doesn't seem to work either (returns an empty SubDataFrame).
 
  Thank you,
  Andrew

Re: [julia-users] ANN: IndexedArrays.jl

[Tom Short]

For something somewhat related, there is AxisArrays, a relatively new
package that isn't registered in METADATA, yet:

https://github.com/mbauman/AxisArrays.jl

It's different in that axes of an array can be indexed into with strings or
symbols. There is a plan to add something like IndexedArray as an axis type
for faster lookups:

https://github.com/mbauman/AxisArrays.jl/issues/7

Also related is some work in Base that may make something like this
possible with the default Dict:

https://github.com/JuliaLang/julia/pull/10116




On Thu, Feb 26, 2015 at 11:43 PM, Jim Garrison j...@garrison.cc wrote:

  I made an `IndexedArrays` package with an `IndexedArray` data structure.
 This acts like a `Vector` of unique elements, but also maintains the
 reverse mapping in a dictionary that is updated in sync with the vector,
 allowing for quick lookup of the index of any element.  A simple example is
 probably more useful:

 julia using IndexedArrays

 julia ia = IndexedArray([cat, dog, mouse])
 3-element IndexedArray{ASCIIString}:
  cat
  dog
  mouse

 julia ia[1]
 cat

 julia findfirst(ia, dog) # executes quickly via a
 dictionary lookup, not sequential search
 2

 More details at https://github.com/garrison/IndexedArrays.jl

 In particular, I am looking for

 - A link if something similar already exists
 - Obligatory bikeshedding on the chosen package name
 - Any thoughts/comments/suggestions!

 Jim

Re: [julia-users] Using a function on a group of data

[Tom Short]

Here are some links that might help:

http://dataframesjl.readthedocs.org/en/latest/split_apply_combine.html

https://github.com/JuliaStats/DataFramesMeta.jl/

https://groups.google.com/forum/#!topic/julia-users/BFW2cvAOfC4


On Tue, Feb 24, 2015 at 6:47 AM, pgs philsiv...@googlemail.com wrote:

 Hi
 I have a txt file which I'm trying to perform a compuation(FUNCTION) on
 groups of data - the Grouping being Col1 then Col2.
 eg

 Say my txt file has the following

 Col1,Col2,Col3 Col4
 A 15:00 4  8
 A 15:00 6  5
 A 15:30 2  4
 A 15:30 8  7
 B 12:00 5  2
 B 12:00 4 1
 B 18:00 2 6
 B 18:00 1 7
 B 18:00 4 0.5  etc etc

 myfunction = map((x,y) - x / y * 2, Col3,Col4)
 I then want to sum the results of my computation using the function e g

 So, when applying myfunction? - the result set is...
 A 15:00 1
 A 15:00 2.4
 sum 3.4
 A 15:30 1
 A 15:30 2.28
 sum 3.28
 So, the end game for me is just to see the final sum as an output - eg
 A 15:00 3.4
 A 15:30 3.28
 etc
 is this possible?

 Regards

Re: [julia-users] Numerical precision of simple mathematical operations

[Tom Short]

This section in the manual might help:

http://julia.readthedocs.org/en/latest/manual/integers-and-floating-point-numbers/#background-and-references

On Thu, Feb 12, 2015 at 3:48 AM, jung.fe...@gmail.com wrote:

 Hi,

 I've run into a problem that I hadn't even considered so far. When
 computing the analytical solution to a positive integral (its an integral
 over a quadratic function) I got extremely negative values from Julia. Here
 is some example code:

 function compute(f1, f2, f3, d, l, u)
 ## Some addends part of a closed form solution of a positive integral.
 s1 = (8 * f1 * (f2 + f3 + d * f3 * l)) / exp(d * l)
 s2 = (2 * f2^2 + 2 * f2 * (f3 + 2 * d * f3 * l) + f3^2 *
 (1 + 2 * d * l + 2 * d^2 * l^2)) / exp(2 * d * l)
 s3 = 4 * d * f1^2 * (-l + u)
 s4 = -(8 * f1 * (f2 + f3 + d * f3 * u)) / exp(d * u)
 s5 = (-2 * f2^2 - 2 * f2 * (f3 + 2 * d * f3 * u) - f3^2 *
 (1 + 2 * d * u + 2 * d^2 * u^2)) / exp(2 * d * u)
 ## Some divisor we are going to use later on
 d = 4 * d


 return s1, s2, s3, s4, s5, d
 end


 x  = [1, 0.045126, 6.9315]
 w1 = w2 = w3 = [1 1 1]
 w4 = [1 -10 -10]

 f1 = w1 * x
 ## 7.97663
 f2 = w2 * x
 ## 7.97663
 f3 = w3 * x
 ## 7.97663
 d  = exp(w4 * x)
 ## 1.36519e-30



 # Compute the addends that go into the closed form solution
 s1, s2, s3, s4, s5, d = compute(f1[1], f2[1], f3[1], d[1], 0, 153)
 ## Result:
 ##
 (1018.0249975020159,318.13281171937996,2.1263884953112284e-25,-1018.0249975020159,
 ##  -318.13281171937996,2.1843022017328403e-29)


 # You can see that s1, s4 and s2, s5 are offsetting, exactly.
 s1 + s4
 ## Result:
 ## s1 + s4 = 0.0
 s2 + s5
 ## Result:
 ## s2 + s5 = 0.0


 # The actual solution we need to compute is
 # (s1 + s2 + s3 + s4 + s5) / (4 * d),
 # which should yield s3 / (4 *d).
 # However, computing s1 + s2 + s3 + s4 + s5 yields a negative number!
 s1 + s2 + s3 + s4 + s5
 ## Result:
 ## -5.684341886080802e-14


 # Hence the whole expression (s1 + s2 + s3 + s4 + s5) / (4 * d) becomes
 negative:
 (s1 + s2 + s3 + s4 + s5) / (4 * d)
 ## Result:
 ## -6.505901383026724e14



 As you can see I get an unacceptable result for an expression that is
 supposed to be positive. You could attribute this to the extremely small
 value of d. Unfortunately, I'm computing this analytical integral as part
 of an MLE without any constraints on the parameter space. I don't want to
 put any constraints in place just to deal with with numerical problems.

 Investigating the issue further, I've noticed that the problem is rooted
 much deeper. Both, in Julia and in R, you can't rely on results in the
 simplest calculations. For example:
 ## In Julia
 julia 1080.02 + 318.2 - 1080.02 - 318.2
 5.684341886080802e-14
 ## In R
  1080.02 + 133.12 - 1080.02 - 133.12
 [1] -1.136868e-13

 I'm aware that floats and real numbers just have these sorts of problems.
 The consequences in my case are just so terrible. I have analytical
 solutions to a log-likelihood function, its gradient and its Hessian, but
 can't trust any of them at the moment, because they might compute to
 garbage.
 What's your experience with this sort of issue? Any advice on how to solve
 this problem?
 Thanks in advance,
 Felix

Re: [julia-users] Input parameters for a simulation: Extract variables from a dictionary?

[Tom Short]

You probably don't want `eval` unless there's no other way. It's hard to
tell how you want to use the variables, so it's hard to recommend
alternatives. Keyword arguments can be useful for this sort of thing:

function f(; a = 1, b = 2)
a + b
end

f(a = 99, b = 2)

You can also use Dict's to pass keyword arguments into functions as
follows:

using Compat
d = @compat Dict(:a = 3, :b = 4)

f(;d...)

Creating a type that holds the parameters is another good option, depending
on how the parameters will be used.


On Wed, Feb 11, 2015 at 3:25 PM, David P. Sanders dpsand...@gmail.com
wrote:

 Hi,

 If I have a dictionary

 params = {N: 10, M: 2.0}

 how can I use it to define two variables N and M with the corresponding
 values?

 This sounds like it should be easy and obvious, say using `eval`?
 E.g. extract the keys and values into strings and then use

 eval(parse(N=10))

 Is this reasonable?

 The use case is to load input parameters for a simulation. Maybe there is
 a better way?

 Thanks,
 David.

Re: [julia-users] @with_fields macro

[Tom Short]

The DataFramesMeta packages has an @with macro that does something similar:

https://github.com/JuliaStats/DataFramesMeta.jl/

It currently works for Associative types and AbstractDataFrames (object
lookups based on getindex). One difference is that you refer to fields
using symbols, so :bar + :baz in your example. You could use that as a
starting point for a macro that works with Composite types. The @with in
DataFramesMeta could probably be extended to work with Composite types. In
@with, the use of getindex would have to be abstracted out, so field lookup
can be used instead.



On Tue, Feb 10, 2015 at 9:45 AM, Tamas Papp tkp...@gmail.com wrote:

 Hi,

 Is there a macro in Julia similar to CL's WITH-SLOTS? Eg

 type Foo
   bar
   baz
 end

 function foo(f::Foo)
   @with_fields f (bar,baz)
 bar+baz
   end
 end

 would be the same as

 function foo(f::Foo)
   f.bar + f.baz
 end

 If not, then some hints on how to write this would be welcome. I could
 not find anything equivalent to symbol macros in Julia, which are used
 to implement this in CL.

 Best,

 Tamas

Re: [julia-users] Re: Sundials CVODE error

[Tom Short]

I've gotten the same error. Here's a proposed fix:

https://github.com/JuliaLang/Sundials.jl/pull/42
On Feb 9, 2015 6:50 PM, Kirill Ignatiev kirill.ignat...@gmail.com wrote:

 On Monday, 9 February 2015 10:49:45 UTC-5, Chris wrote:

 Additionally, I just tried running my code on my Linux VM, and it works
 fine there as well, so for me this is Windows-specific.


 That sounds like memory corruption, could be a mismatch between some
 types, like Int/Int32/Cint/Clong, because IIRC those can be different on
 Windows. Perhaps file this issue with Sundials.jl on github?

Re: [julia-users] using Docile

[Tom Short]

Try installing Lexicon and load it then repeat your example.

On Thursday, February 5, 2015, Diego Tapias dandrove...@gmail.com wrote:

 Hi guys,

 I didn’t have problems installing Docile, however when I try to use it
 literally nothing happens. Before continue reading, my julia version is
 0.4.0-dev.

 Consider the following code :

 filename test.jl

 using Docile

 export foo

 @doc This function is not important -

 function foo()

 end

 Now I type in Julia console:

 include(“test.jl”)

 ?foo

 foo (generic function with 1 method)

 And nothing about documentation.

 ​

Re: [julia-users] ANN: Docile Lexicon update.

[Tom Short]

You'll need to install Lexicon (I think).

On Sat, Jan 31, 2015 at 3:57 PM, i.costi...@me.com wrote:

 I installed Sims and ran `?sim` at REPL. I got the following back:
 `dasslsim (generic function with 20 methods)`. I see that there is an
 extensive docstring against sim in src/sim.jl. Why am I not seeing this at
 REPL? I'm on julia 0.3.5

 On Thursday, 22 January 2015 03:54:14 UTC+11, Ivan Ogasawara wrote:

 great!
 El 21/01/2015 14:22, Michael Hatherly michael...@gmail.com escribió:

 Hi all,

 I’m pleased to announce the latest update to the Docile
 https://github.com/MichaelHatherly/Docile.jl and Lexicon
 https://github.com/MichaelHatherly/Lexicon.jl documentation packages.

 New features include:

- Docile now supports plain strings
http://michaelhatherly.github.io/Docile.jl/syntax/#plain-strings,
ie. without @doc, as docstrings. Compatibility with the Julia 0.4
doc system is still present.
- Thanks to Tom Short, Lexicon can now output nicely formatted
markdown. This can then be used to create static documentation using
programs such as MkDocs http://www.mkdocs.org/. See the
documentation from the following packages for examples of the results:
Sims https://tshort.github.io/Sims.jl/, Docile
http://michaelhatherly.github.io/Docile.jl/, and Lexicon
http://michaelhatherly.github.io/Lexicon.jl/.

 Any bugs or feature requests can be opened in either the Docile or
 Lexicon repos.

 Happy documenting!

 — Mike
 ​

Re: [julia-users] Convention for naming types and functions

[Tom Short]

http://julia.readthedocs.org/en/latest/manual/style-guide/#use-naming-conventions-consistent-with-julia-s-base

On Thu, Jan 29, 2015 at 12:56 PM, Diego Tapias dandrove...@gmail.com
wrote:

 Hello,

 can you please help me with the convention for naming created types
 and functions?

 Diego.

Re: [julia-users] Re: I cannot fine the code for Garch in mean

[Tom Short]

Might be a good starting point:

https://github.com/AndreyKolev/GARCH.jl



On Tue, Jan 27, 2015 at 5:41 PM, colintbow...@gmail.com wrote:

 Hi Jase,

 Good to see you on Julia. Regarding GarchM, I'm fairly certain this
 doesn't exist yet. As Andreas has said, there isn't much time-series
 econometrics on Julia yet at all. I'm planning on adding some capabilities
 over the next few months, e.g. dependent bootstraps and hopefully some
 ARIMA functions, but in general it will probably be a fairly slow process.
 For now, if you're after pre-packaged conditional volatility models, you
 will still probably need R or Kevin Sheppard's Matlab toolbox. If you want
 to work in Julia, you can call R functions using the following package:
 https://github.com/lgautier/Rif.jl

 Perhaps this is an option?

 Cheers mate,

 Colin

 On Tuesday, 27 January 2015 16:17:01 UTC+11, Jung Soo Park wrote:


 Is there any way I can have the code for Garch in Mean or GarchM please

 Thank you..

Re: [julia-users] Saving Julia dataframe to read in R using HDF5

[Tom Short]

I don't know if it can do it yet, but the RCall package might be able to
save data back to an RData file. It's a young package.

Also, you could use CSV files.

On Thu, Jan 22, 2015 at 2:09 PM, Pavel pavel.paramo...@gmail.com wrote:

 While reading R datasets in Julia received sufficient attention already,
 sometimes the results of computations done in Julia need to be readable to
 R. To accomplish that I was trying to save a DataFrame.jl
 https://github.com/JuliaStats/DataFrames.jl object in HDF5 file. The
 code so far is in my StackOverflow question (probably should have posted
 here instead):

 http://stackoverflow.com/questions/28084403/saving-julia-dataframe-to-read-in-r-using-hdf5

 The dataframe can then be reassembled in R using rhdf5
 http://www.bioconductor.org/packages/release/bioc/html/rhdf5.html package
 tools. It works in principle, but is there a more elegant way to accomplish
 this? Something that does not require to split the dataframe apart and
 re-assemble in R, losing some column types (e.g. boolean does not work)
 along the way?

Re: [julia-users] Re: Plotting-package-independent API

[Tom Short]

Sims.jl uses the Requires.jl approach that Matt mentioned. There are a few
helper plotting methods that are optional depending on what packages the
user loads:

https://github.com/tshort/Sims.jl/blob/master/src/utils.jl#L158-L181

https://tshort.github.io/Sims.jl/plotting/



On Thu, Jan 22, 2015 at 3:36 PM, Matt Bauman mbau...@gmail.com wrote:

 I've thought some about this in the past, too.  Coming from Matlab, I've
 imagined a Matlab-esque Base.Plots module that has just a handful of
 definitions, mirroring the IO system:

 abstract AbstractPlot
 immutable NoPlots : AbstractPlot end
 STDPLOT = NoPlots() # When a plotting package loads, it assigns its own
 AbstractPlot object to STDPLOT

 line(args...) = line(STDPLOT, args...)
 line(p::AbstractPlot, ys::AbstractArray) = line(p, 1:size(ys,1), ys)
 # Then each plotting package would specialize on the following method:
 line(::NoPlots, xs, ys) = error(Plotting requires a plotting package to
 be installed.  Here, try one of these: …)

 # … and so on for just a handful of very common (and specific!) names …
 scatter() …
 bar() …
 surface() …

 Of course, doing anything more complicated would require delving into
 documentation and using a more package-specific function.  But users could
 easily start from the source for these simple prototypes and grow from
 there.  Sure, this will be heretical for some plotting packages, but I
 think there's value here, especially for quick and dirty exploratory data
 analysis.

 The plot function is tougher — there's not really a meaning to it beyond
 show this thing graphically, and packages use keywords or other arguments
 to determine the kind of plot.  I could see it being useful in cases like
 yours, though, where you want to define a plotting routine for a your own
 ROC object type.  You would simply define `plot(p::AbstractPlot, t::
 ROCType)` and then use the above primitives.  The trouble is, as a
 package author, you'll want to add fancier features like labels, legends,
 colors, titles, bounds, error-bars, shaded regions, transforms, etc…. and
 now we're marching down that crazy never-ending path Daniel fears.

 So: yes, it's possible.  Is it useful? I'm not so sure.  I think no matter
 what you're going to have to learn a specific plotting API, and as a
 package author I think you're better off picking one to support well and
 extending its functions directly.  Maybe some cavalier users will come by
 and add support for others they like with lazy loading through Requires.jl.
 :)

 Matt

 On Thursday, January 22, 2015 at 1:03:44 PM UTC-5, Simon Danisch wrote:

 The reference does seem to fit here, as a general plotting language seems
 to be awfully complicated.
 By the way, I totally forgot to mention, that I want to turn GLPlot more
 into a meta package, offering different plotting interfaces, since I'm
 generalizing more and more of the render code and a lot of people prefer
 different interfaces to the same functionality.
 So it might become a lightweight package, which actually doesn't contain
 render code and only defines interfaces.
 If someone has any API that he wants to implement, GLPlot might become
 the go to package. If this happens, I should rename it though ;)


 Am Donnerstag, 22. Januar 2015 16:42:37 UTC+1 schrieb David van Leeuwen:

 Hello,

 I've just read up on most recent conversations mentioning plotting,
 after I went through a similar install-and-try-out cycle for the various
 plotting packages.

 I am busy with a package named ROC https://github.com/davidavdav/ROC 
 (Receiver
 Operating Characteristic).  It is mostly about decision cost functions and
 efficiently computing ROC statistics, but there is also some support for
 making various plots related to the ROC.

 I do not want to decide for a potential user of this package which
 plotting package she should use.  So it would be nice if there would be an
 abstract plotting API (I suppose this does not exist yet).

 Supposing that it is possible to make an API that includes some common
 plotting primitives, and that makes a runtime selection based on 
 `isdefined(plottingpackagename)`,
 how would you
  - overload a function, typically `plot()`, when at module-include time
 it is not yet known which plotting package the user wants
  - specify the dependence of this package to any one of the available
 plotting packages.

 Thoughts?

 Thanks,

 ---david

Re: [julia-users] exemplar package for 0.4-style documentation

[Tom Short]

It just hasn't been implemented, yet. Lexicon complains with type
BasicREPL has no field interface. I'm guessing that a simpler approach
could be used in Lexicon, or it could be implemented in ESS (unfortunately,
no ESS developers really use Julia).

On Wed, Jan 14, 2015 at 4:23 AM, Tamas Papp tkp...@gmail.com wrote:

 On Tue, Jan 13 2015, Tom Short tshort.rli...@gmail.com wrote:

  If you want to try the Sims docs at the REPL, you'll need to checkout
 that
  package and enter using Sims, Sims.Lib, Lexicon. Then you can do
  ?Resistor or ?Capacitor.
 
  Note that the question mark doesn't work in ESS.

 Is that an intrinsic limitation of the documentation system/julia/ESS,
 or just something that has not been implemented yet? It would be very
 nice, since other languages (eg R) use ?function too and that works
 nicely in ESS so users are used to it.

 Best,

 Tamas

Re: [julia-users] Naming of functions advice

[Tom Short]

Another option is to do both! You could write methods in your main Controls
module done the Julian way with appropriate use of method dispatch. Then,
you could add a Controls.Compat submodule where you define methods that
follow the Matlab api.
On Jan 12, 2015 2:09 PM, James Crist crist...@umn.edu wrote:

 I'm currently writing a control theory package for Julia. The MATLAB
 control theory toolbox has been around forever, and is *the standard*
 grammar for the field. Almost all control theory packages in other
 languages just replicate the same set of functions, with the same names.
 The function names are so hardcoded into me, that I'm reluctant to change
 them.

 That said, several of them conflict with Julia base function names. For
 example:

 `zero(sys::LTISystem)` would compute the zeros of the system in MATLAB,
 but in Julia this should create the zero-valued system

 `step(sys::LTISystem)` computes the step-response, but in julia it gives
 the step for a range

 There are others as well. I see two options here:

 1.) I begrudgingly rename them, and attempt to retrain my muscle-memory
 when writing code :/
 2.) Some functions don't do what they do in julia base for these types

 #1 probably is for the best, but I'm wondering what the community response
 is to this? I come from a heavy Python background, and without namespaces,
 I'm not sure how to handle function name-clashing best.

Re: [julia-users] Is there a natural way to use the same docstring for multiple functions?

[Tom Short]

It's not what you are asking, but Lexicon/Docile has a couple of nice
features for grouping doc strings. First, to document a generic function
before it's defined, you can do:


My doc string here

:genericfun

genericfun(x::something1) = 1
genericfun(x::something2) = 2


You can also define a docstring that applies to a set of methods as:

My doc string here

(getindex, :MyType, Any...)

Both are good ways to group functions you define.







On Tue, Jan 13, 2015 at 4:00 PM, Gray Calhoun gcalh...@iastate.edu wrote:

 Oh, perfect. The @doc (@doc ) construction is exactly what I was looking
 for. Thanks!

 On Tuesday, January 13, 2015 at 2:35:09 PM UTC-6, Mike Innes wrote:

 That should pretty much work as is, but you'll want to use the `doc`
 string macro for markdown formatting, i.e.

 recursive_ols_doc = doc 

 The other thing you can do is document one function, then retrieve that
 functions docs with the @doc macro:

 @doc foo -
 function recursive_ols() ...

 @doc (@doc recursive_ols) -
 function recursive_ols!() ...

 On 13 January 2015 at 19:49, Gray Calhoun gcal...@iastate.edu wrote:

 Hi everyone, quick question about docstrings. Sometimes it makes sense
 to use the same help text for different functions; I'm particularly
 thinking mutating and non-mutating versions of the same algorithm. Is
 there a natural way to reuse a docstring for different functions
 (either in Docile.jl or in v0.4 base)?

 Specifically, I'm doing something like the following, but would prefer
 to avoid the temporary variable `recursive_ols_doc` if possible (for
 style reasons, if nothing else), and without writing the documentation in
 a separate file.

 recursive_ols_doc = 
   Calculates the OLS estimators for the model y ~ x recursively...
   * `recursive_ols` is a wrapper that works like this...
   * `recursive_ols!` does all the real work like that...
 

 @doc recursive_ols_doc -
 function recursive_ols!(estimates::Array, y, x)
 ## Calculations
 end

 @doc recursive_ols_doc -
 function recursive_ols(y, x, R::Integer)
 ## preallocate, then call recursive_ols!
 end

 Thanks!

 (Really liking this documentation tool, btw)

Re: [julia-users] exemplar package for 0.4-style documentation

[Tom Short]

It's still a bit in flux as the old style is:

@doc 
my docs...
 -
myfun(x) = ...

The new style is:


my docs...

myfun(x) = ...

The new style is in Docile but not in v0.4-, yet.

The documentation inside Docile is a good start. It uses the new style.
Some other notable packages include:

* https://github.com/johnmyleswhite/CSVReaders.jl

* https://tshort.github.io/Sims.jl/

In Sims.jl, I used Mkdocs (mkdocs.org) along with Docile/Lexicon to create
that documentation site.






On Tue, Jan 13, 2015 at 4:59 PM, Douglas Bates dmba...@gmail.com wrote:

 I'm having difficulty navigating my way around documentation standards for
 v0.4-, and whether the Docile or Lexicon or Markdown packages are
 needed.  What would be a good package to study to see how @doc, etc. should
 be used?

Re: [julia-users] exemplar package for 0.4-style documentation

[Tom Short]

The source with doc strings was only recently merged into master. They
should be there now (but it's not in METADATA, yet). Compare the input and
output for one of the source files:

* https://github.com/tshort/Sims.jl/blob/master/examples/lib/electrical.jl

* https://tshort.github.io/Sims.jl/examples/lib/#electrical

On Tue, Jan 13, 2015 at 5:14 PM, Douglas Bates dmba...@gmail.com wrote:

 Actually the first place I looked was in your Sims package but I didn't
 see any documentation strings, of either form, in the Julia sources.  Am I
 missing something very basic here?

 On Tuesday, January 13, 2015 at 4:06:44 PM UTC-6, tshort wrote:

 It's still a bit in flux as the old style is:

 @doc 
 my docs...
  -
 myfun(x) = ...

 The new style is:

 
 my docs...
 
 myfun(x) = ...

 The new style is in Docile but not in v0.4-, yet.

 The documentation inside Docile is a good start. It uses the new style.
 Some other notable packages include:

 * https://github.com/johnmyleswhite/CSVReaders.jl

 * https://tshort.github.io/Sims.jl/

 In Sims.jl, I used Mkdocs (mkdocs.org) along with Docile/Lexicon to
 create that documentation site.






 On Tue, Jan 13, 2015 at 4:59 PM, Douglas Bates dmb...@gmail.com wrote:

 I'm having difficulty navigating my way around documentation standards
 for v0.4-, and whether the Docile or Lexicon or Markdown packages are
 needed.  What would be a good package to study to see how @doc, etc. should
 be used?

Re: [julia-users] exemplar package for 0.4-style documentation

[Tom Short]

If you want to try the Sims docs at the REPL, you'll need to checkout that
package and enter using Sims, Sims.Lib, Lexicon. Then you can do
?Resistor or ?Capacitor.

Note that the question mark doesn't work in ESS.



On Tue, Jan 13, 2015 at 5:21 PM, Tom Short tshort.rli...@gmail.com wrote:

 The source with doc strings was only recently merged into master. They
 should be there now (but it's not in METADATA, yet). Compare the input and
 output for one of the source files:

 * https://github.com/tshort/Sims.jl/blob/master/examples/lib/electrical.jl

 * https://tshort.github.io/Sims.jl/examples/lib/#electrical

 On Tue, Jan 13, 2015 at 5:14 PM, Douglas Bates dmba...@gmail.com wrote:

 Actually the first place I looked was in your Sims package but I didn't
 see any documentation strings, of either form, in the Julia sources.  Am I
 missing something very basic here?

 On Tuesday, January 13, 2015 at 4:06:44 PM UTC-6, tshort wrote:

 It's still a bit in flux as the old style is:

 @doc 
 my docs...
  -
 myfun(x) = ...

 The new style is:

 
 my docs...
 
 myfun(x) = ...

 The new style is in Docile but not in v0.4-, yet.

 The documentation inside Docile is a good start. It uses the new style.
 Some other notable packages include:

 * https://github.com/johnmyleswhite/CSVReaders.jl

 * https://tshort.github.io/Sims.jl/

 In Sims.jl, I used Mkdocs (mkdocs.org) along with Docile/Lexicon to
 create that documentation site.






 On Tue, Jan 13, 2015 at 4:59 PM, Douglas Bates dmb...@gmail.com wrote:

 I'm having difficulty navigating my way around documentation standards
 for v0.4-, and whether the Docile or Lexicon or Markdown packages are
 needed.  What would be a good package to study to see how @doc, etc. should
 be used?

Re: [julia-users] How to define help text for a module function so that it displays in REPL

[Tom Short]

Docile.jl works great. With Lexicon (or base Julia 0.4dev), you get nice
REPL help.

Mkdocs (http://mkdocs.org) works great for taking Markdown from Docile
output along with other Markdown files to make online documentation. Mkdocs
is super easy and works great with Github Pages. It also works with
readthedocs (but I haven't tried that).

Here is an example of package documentation developed with Docile and
Mkdocs (still a work in progress):

https://tshort.github.io/Sims.jl/



On Fri, Jan 9, 2015 at 3:41 AM, Ján Dolinský jan.dolin...@2bridgz.com
wrote:

 Hi,

 I would like to know how to write a help (comment) text e.g. for a module
 function so that once module is loaded it is displayed if in REPL help mode.
 E.g. for sum function
 help?sum
 displays brief help text for sum

 Thanks,
 Jan

Re: [julia-users] How to define help text for a module function so that it displays in REPL

[Tom Short]

Sorry, it's hidden in a branch (again, a work in progress):

* https://github.com/tshort/Sims.jl/tree/mkdocs

Here is the documentation directory with the static markdown files and the
markdown files generated from Docile:

* https://github.com/tshort/Sims.jl/tree/mkdocs/docs

Compare the input and output for one of the source files:

* https://github.com/tshort/Sims.jl/blob/mkdocs/examples/lib/electrical.jl:

* https://tshort.github.io/Sims.jl/examples/lib/#electrical





On Sat, Jan 10, 2015 at 12:06 PM, Petr Krysl krysl.p...@gmail.com wrote:

 Hi,

 I had a look at your package: very nice indeed!  It is not clear to me
 where the actual documentation is though.  I followed some links to source
 code and there was no documentation in there…

 Petr

 On Saturday, January 10, 2015 at 8:35:24 AM UTC-8, tshort wrote:

 Docile.jl works great. With Lexicon (or base Julia 0.4dev), you get nice
 REPL help.

 Mkdocs (http://mkdocs.org) works great for taking Markdown from Docile
 output along with other Markdown files to make online documentation. Mkdocs
 is super easy and works great with Github Pages. It also works with
 readthedocs (but I haven't tried that).

 Here is an example of package documentation developed with Docile and
 Mkdocs (still a work in progress):

 https://tshort.github.io/Sims.jl/



 On Fri, Jan 9, 2015 at 3:41 AM, Ján Dolinský jan.do...@2bridgz.com
 wrote:

 Hi,

 I would like to know how to write a help (comment) text e.g. for a
 module function so that once module is loaded it is displayed if in REPL
 help mode.
 E.g. for sum function
 help?sum
 displays brief help text for sum

 Thanks,
 Jan

Re: [julia-users] How to define help text for a module function so that it displays in REPL

[Tom Short]

Also, the links to source code don't work (I think) because I'm working on
a branch. They should work once I merge to master.

The only issue I found with Mkdocs was with syntax highlighting. Just a bit
of googling, though. I had to use the codehighlight extension that uses
Pygments. I also needed to add a css file. See the mkdocs.yml file.

On Sat, Jan 10, 2015 at 12:42 PM, Tom Short tshort.rli...@gmail.com wrote:

 Sorry, it's hidden in a branch (again, a work in progress):

 * https://github.com/tshort/Sims.jl/tree/mkdocs

 Here is the documentation directory with the static markdown files and the
 markdown files generated from Docile:

 * https://github.com/tshort/Sims.jl/tree/mkdocs/docs

 Compare the input and output for one of the source files:

 * https://github.com/tshort/Sims.jl/blob/mkdocs/examples/lib/electrical.jl
 :

 * https://tshort.github.io/Sims.jl/examples/lib/#electrical





 On Sat, Jan 10, 2015 at 12:06 PM, Petr Krysl krysl.p...@gmail.com wrote:

 Hi,

 I had a look at your package: very nice indeed!  It is not clear to me
 where the actual documentation is though.  I followed some links to source
 code and there was no documentation in there…

 Petr

 On Saturday, January 10, 2015 at 8:35:24 AM UTC-8, tshort wrote:

 Docile.jl works great. With Lexicon (or base Julia 0.4dev), you get nice
 REPL help.

 Mkdocs (http://mkdocs.org) works great for taking Markdown from Docile
 output along with other Markdown files to make online documentation. Mkdocs
 is super easy and works great with Github Pages. It also works with
 readthedocs (but I haven't tried that).

 Here is an example of package documentation developed with Docile and
 Mkdocs (still a work in progress):

 https://tshort.github.io/Sims.jl/



 On Fri, Jan 9, 2015 at 3:41 AM, Ján Dolinský jan.do...@2bridgz.com
 wrote:

 Hi,

 I would like to know how to write a help (comment) text e.g. for a
 module function so that once module is loaded it is displayed if in REPL
 help mode.
 E.g. for sum function
 help?sum
 displays brief help text for sum

 Thanks,
 Jan

Re: [julia-users] Package name for embedding R within Julia

[Tom Short]

I like the name RCall.

Maybe you could have an R module inside of RCall, so you can use R.plot,
R.sum, and bring in as much as possible without worrying about namespaces.

Also, you might want to use the PyCall code and general interface for
guidance.



On Fri, Jan 2, 2015 at 2:59 PM, Douglas Bates dmba...@gmail.com wrote:

 For many statistics-oriented Julia users there is a great advantage in
 being able to piggy-back on R development and to use at least the data sets
 from R packages.  Hence the RDatasets package and the read_rda function in
 the DataFrames package for reading saved R data.

 Over the last couple of days I have been experimenting with running an
 embedded R within Julia and calling R functions from Julia. This is similar
 in scope to the Rif package except that this code is written in Julia and
 not as a set of wrapper functions written in C. The R API is a C API and,
 in some ways, very simple. Everything in R is represented as a symbolic
 expression or SEXPREC and passed around as pointers to such expressions
 (called an SEXP type).  Most functions take one or more SEXP values as
 arguments and return an SEXP.

 I have avoided reading the code for Rif for two reasons:
  1. It is GPL3 licensed
  2. I already know a fair bit of the R API and where to find API function
 signatures.

 Here's a simple example
 julia initR()
 1

 julia globalEnv = unsafe_load(cglobal((:R_GlobalEnv,libR),SEXP),1)
 Ptr{Void} @0x08c1c388

 julia formaldehyde = tryEval(install(:Formaldehyde))
 Ptr{Void} @0x08fd1d18

 julia inherits(formaldehyde,data.frame)
 true

 julia printValue(formaldehyde)
   carb optden
 1  0.1  0.086
 2  0.3  0.269
 3  0.5  0.446
 4  0.6  0.538
 5  0.7  0.626
 6  0.9  0.782

 julia length(formaldehyde)
 2

 julia names(formaldehyde)
 2-element Array{ASCIIString,1}:
  carb
  optden

 julia form1 = ccall((:VECTOR_ELT,libR),SEXP,(SEXP,Cint),formaldehyde,0)
 Ptr{Void} @0x0a5baf58

 julia ccall((:TYPEOF,libR),Cint,(SEXP,),form1)
 14

 julia carb =
 copy(pointer_to_array(ccall((:REAL,libR),Ptr{Cdouble},(SEXP,),form1),length(form1)))
 6-element Array{Float64,1}:
  0.1
  0.3
  0.5
  0.6
  0.7
  0.9

 julia form2 = ccall((:VECTOR_ELT,libR),SEXP,(SEXP,Cint),formaldehyde,1)
 Ptr{Void} @0x0a5baef0

 julia ccall((:TYPEOF,libR),Cint,(SEXP,),form2)
 14

 julia optden =
 copy(pointer_to_array(ccall((:REAL,libR),Ptr{Cdouble},(SEXP,),form2),length(form2)))
 6-element Array{Float64,1}:
  0.086
  0.269
  0.446
  0.538
  0.626
  0.782


 A call to printValue uses the R printing mechanism.

 Questions:
  - What would be a good name for such a package?  In the spirit of PyCall
 it could be RCall or Rcall perhaps.

  - Right now I am defining several functions that emulate the names of
 functions in R itself ir in the R API.  What is a good balance?  Obviously
 it would not be a good idea to bring in all the names in the R base
 namespace.  On the other hand, those who know names like inherits and
 what it means in R will find it convenient to have such names in such a
 package.

 - Should I move the discussion the the julia-stats list?

Re: [julia-users] Slow Devectorized Computation in Julia

[Tom Short]

It should probably be the 1st faq. It's the first performance tip though:

http://julia.readthedocs.org/en/latest/manual/performance-tips/

Put your code in functions (or liberally use const). Globals are slow.
On Dec 27, 2014 10:24 PM, Bob Quazar reg...@gmail.com wrote:

 The devectorized code below should be much faster than equivalent
 vectorized code, according to Fast Numeric Computation in Julia
 http://julialang.org/blog/2013/09/fast-numeric/ on the official
 julialang.org web site. But I find just the opposite!

 julia x = randn(1000);
 julia y = randn(1000);
 julia r = zeros(1000);
 julia @time r[:] = exp(-abs(x-y));
 elapsed time: 1.233813455 seconds (32432 bytes allocated, 14.20% gc
 time)
 julia @time for i = 1:length(x)
r[i] = exp(-abs(x[i]-y[i]))
end
 elapsed time: 10.326934093 seconds (1599983704 bytes allocated, 21.29% gc
 time)


 Can someone explain what's going on? The devectorized version appears to
 allocate 5x as much memory.

 Thanks much!

Re: [julia-users] Converting a dictionary object to a Data Frame

[Tom Short]

Try this:

convert(DataFrame, d)   # where d is your dict

You can also use a more long-winded direct call:

DataFrame(collect(values(d)), collect(keys(d)))





On Sun, Dec 21, 2014 at 10:28 PM, Min-Woong Sohn mws...@gmail.com wrote:

 Is there a way to convert a dictionary into a data frame in such a way
 that the keys in the dictionary become indexes in the data frame and values
 become a column?

Re: [julia-users] Re: Can Julia be the next web scripting language?

[Tom Short]

It's getting closer. First, the --compile=all and --dump-bitcode=yes
options to Julia are a big help in dumping LLVM bitcodes. The
contrib/build_sysimg.jl code is useful for this after modifying it to
include these Julia command-line options.

There are a few issues with using Emscripten with Julia code:

* 32-bit code -- Emscripten expects 32-bit bitcode. On a 64-bit platform,
Julia generates 64-bit bitcode. I tried using the build_sysimg.jl script
with a target of i686, but Emscripten still gave me errors.

* struct error -- See this bug:

https://code.google.com/p/nativeclient/issues/detail?id=3932

* Symbol naming -- Code generated by Julia has a lot of symbols like #sym
that Emscripten doesn't like. This can be handled by either Emscripten or
on the Julia side.

Lastly, libjulia bitcode needs to be generated. I've gotten about 80% of it
to compile after stripping out the libuv stuff. There are also libraries
like BLAS, but these can be added as needed.





On Thu, Dec 18, 2014 at 2:48 PM, Páll Haraldsson pall.haralds...@gmail.com
wrote:


 On Thursday, December 18, 2014 10:59:53 AM UTC, Jerzy Głowacki  wrote:

 I suppose Julia would be the successor to JavaScript if it was converted
 by Emscripten and run by asm.js. But for now there seems to be an
 Enscripten bug which invalidates Julia IR code.


 Hi,

 Curious, what bug is that? You have a bug number or know what it relates
 to (on Emscripten side)? Invalidates Julia IR code? Intermediate
 Representation? Does the bug only affect Julia? Have anything to do with
 JavaScript ints (that it their non-existence, JS uses doubles as int)?

 Did you really mean to say Julia would be the successor or Julia could
 be a successor?

 Is there any obvious successor or fairly much used alternative
 currently? I've heard about Coffeescript, Typescript and Dart etc.
 languages made to compile to JavaScript. And Scala.js. Scala was meant to
 run in JVM and Scala.js has slightly different semantics because if JS's
 ints. I wandered if Julia could do the same.

 I assume you are talking about compile Julia the language/environment to
 JavaScript and run the the REPL in a browser?

 Should it be easier to compile Julia code to C (possible now) and that C
 code via Emscripten to JavaScript? Emscripten says they handle portable C
 code. Doesn't Intel's Julia2C claim portable C? I don't really see how
 Emscripten handles ints. There is no requirement that C's ints are less
 than 64 bit? Do not even have to be two's compliment, right? I fail to see
 how they densely pack ints (that are 64-bit doubles) into 32-bit in
 arrays..


 PS. I dreamt my first dream in Julia code tonight. Might be a sign it's
 the language for me.. :) Reminds me: Dreaming in code is a good book on
 (Python and) Chandler (and e.g. Kapor/Mozilla)..

 PS2. Is there an alternative gateway to the forums. I just hate using the
 web(browser it's slow). Google Groups used to be for Usenet, but not this
 forum..

Re: [julia-users] How do I add a column to a subdataframe?

[Tom Short]

Tony, it's not possible to manipulate a subdataframe like that with the
current code. You can make something that works close, though. In Julia
v0.4, you can create a DataFrame with columns that are views into the
original columns using `sub`. Here is an example:

df = DataFrame(a = rand(1:3, 10), x = rand(10), y = rand(10))
idx = find(df[:a] .== 1)
sd = DataFrame(Any[sub(df[i], idx) for i in 1:size(df, 2)], names(df))

Try `dump(sd)` to view the structure of the result.

Now, you can use `sd` like a SubDataFrame, but you can do any normal
DataFrame manipulation. If the starting point is a SubDataFrame, you can
convert it as follows:

sdf = sub(df, df[:a] .== 1)
sd2 = DataFrame(Any[sub(sdf.parent[i], idx) for i in 1:size(sdf, 2)],
names(sdf))

One huge improvement of this is approach is that `sd[:colA]` doesn't
allocate anything. With the existing SubDataFrames implementation, a column
indexing operation like that does a row indexing operation into the parent
DataFrame, so it will allocate memory. That differs from a DataFrame where
column indexing never allocates anything.

I've been thinking that this might be a good change in general for `sub`.
If Base Arrays switch to views by default, this might match better with
that. In fact, we could get rid of `sub` and do this for all row indexing
operations. It could use some more thought.

This approach doesn't work in Julia v0.3 because `sub` did not support
arbitrary indexes in v0.3.




On Fri, Dec 12, 2014 at 1:58 AM, Tony Fong tony.hf.f...@gmail.com wrote:


 In a vanilla dataframe, I can do this
 df[ :a ] = mydarray

 It doesn't seem to allow me to do the same for a subdataframe returned by
 a groupby.

 I'm trying to compute a new column based on the subdataframe, attach this
 column to it and then do further groupby.

 Is there a way to do that?

 Tony

Re: [julia-users] [WIP] CSVReaders.jl

[Tom Short]

Exciting, John! Although your documentation may be very sparse, the code
is nicely documented.

On Mon, Dec 8, 2014 at 12:35 AM, John Myles White johnmyleswh...@gmail.com
wrote:

 Over the last month or so, I've been slowly working on a new library that
 defines an abstract toolkit for writing CSV parsers. The goal is to provide
 an abstract interface that users can implement in order to provide
 functions for reading data into their preferred data structures from CSV
 files. In principle, this approach should allow us to unify the code behind
 Base's readcsv and DataFrames's readtable functions.

 The library is still very much a work-in-progress, but I wanted to let
 others see what I've done so that I can start getting feedback on the
 design.

 Because the library makes heavy use of Nullables, you can only try out the
 library on Julia 0.4. If you're interested, it's available at
 https://github.com/johnmyleswhite/CSVReaders.jl

 For now, I've intentionally given very sparse documentation to discourage
 people from seriously using the library before it's officially released.
 But there are some examples in the README that should make clear how the
 library is intended to be used.

  -- John

Re: [julia-users] Re: Easy way to copy structs.

[Tom Short]

You can also define your own constructor that works the same as the issue
Tobias pointed to:

immutable Point
x::Float64
y::Float64
z::Float64
color::Int
collision::Bool
end

Point(pt::Point; x = pt.x, y = pt.y, z = pt.z, color = pt.color, collision
= pt.collision) = Point(x, y, z, color, collision)

setX(pt::Point, x::Float64) = Point(pt, x = x)


On Mon, Dec 8, 2014 at 4:16 PM, Tobias Knopp tobias.kn...@googlemail.com
wrote:

 please have a look at https://github.com/JuliaLang/julia/pull/6122 which
 looks like the way to go (once merged)

 Cheers

 Tobi

 Am Montag, 8. Dezember 2014 20:18:18 UTC+1 schrieb Josh Langsfeld:

 On Monday, December 8, 2014 12:15:23 PM UTC-5, Utkarsh Upadhyay wrote:

 I have just tried playing with Julia and I find myself often copying
 `immutable` objects while changing just one field:

 function setX(pt::Point, x::Float64)
   # Only changing the `x` field
   return Point(x, pt.y, pt.z, pt.color, pt.collision, pt.foo, pt.bar);
 end

 Is there is any syntactical sugar to avoid having to write out all the
 other fields which are not changing (e.g. pt.y, pt.z, etc.) while creating
 a new object?
 Apart from making the function much easier to understand, this will also
 make it much less of a hassle to change fields on the type.

 Thanks.


 I played with it some more and this seems to work:

 defn = setdiff(names(Point), [:x])
  @eval Point(x, $([:(pt.$n) for n in defn]...))

 Of course, this will only work for the x parameter which comes first in
 the field list. I'm not sure how you might extend it to arguments in the
 middle of the list, but maybe you can figure it out.

Re: [julia-users] Re: DataFrames and NamedArrays: are they suitable for heavier computations ?

[Tom Short]

For DataFrames, it depends on what you want to do. It is difficult to get
performance with DataArrays as columns using the current implementation.
With the ongoing work by John Myles White on the use of a Nullable type,
that should be much better. Also, you can use standard Arrays as columns of
a DataFrame. It's not documented well, but it can be done.

Also, if you want to treat a DataFrame like a matrix, then generally the
answer is no. With some trickery, you can store a view to a matrix in a
DataFrame. Basically, you have to create column views into the matrix. Here
is an example. It might be useful if you want to treat all or part of a
DataFrame as a matrix.

julia using DataFrames

julia m = rand(5,5)
5x5 Array{Float64,2}:
 0.186736  0.247699   0.0968634  0.471383   0.145244
 0.985306  0.966015   0.663865   0.0468244  0.0471465
 0.981947  0.707241   0.0841202  0.0539529  0.692217
 0.918222  0.0415162  0.646298   0.581983   0.653881
 0.515692  0.0344289  0.0821672  0.877258   0.653756

julia d = DataFrame(Any[sub(m, :, i) for i in 1:size(m, 2)], [:a, :b, :c,
:d, :e])
5x5 DataFrames.DataFrame
| Row | a| b | c | d | e |
|-|--|---|---|---|---|
| 1   | 0.186736 | 0.247699  | 0.0968634 | 0.471383  | 0.145244  |
| 2   | 0.985306 | 0.966015  | 0.663865  | 0.0468244 | 0.0471465 |
| 3   | 0.981947 | 0.707241  | 0.0841202 | 0.0539529 | 0.692217  |
| 4   | 0.918222 | 0.0415162 | 0.646298  | 0.581983  | 0.653881  |
| 5   | 0.515692 | 0.0344289 | 0.0821672 | 0.877258  | 0.653756  |

julia d[:b]
5-element SubArray{Float64,1,Array{Float64,2},(Colon,Int64),2}:
 0.247699
 0.966015
 0.707241
 0.0415162
 0.0344289



On Fri, Dec 5, 2014 at 8:37 AM, Tim Holy tim.h...@gmail.com wrote:

 The new SubArrays steal mercilessly from the good ideas of both the old
 SubArray and ArrayViews, and then add some new tricks of their own. In
 theory,
 they should be a strict improvement on both of their predecessors.

 --Tim

 On Friday, December 05, 2014 05:13:28 AM David van Leeuwen wrote:
  So what is the relation between ArrayViews and 0.4 `SubArray revamp'?
  Are
  they targeting different use cases or is one of them going to be phased
  out?
 
  On Friday, December 5, 2014 1:28:16 PM UTC+1, Tim Holy wrote:
   In 0.4, the views are a revamp of SubArray. So if NamedArrays already
   interacts well with SubArrays, you're basically set.
  
   Most likely not, as SubArrays are new to me (I've tried sub() in
 
  production code in the past, but it was always better to write out a
  loop)).
 
   FYI the implementation in 0.4 is largely backwards-compatible, but
 there
   are
   some important differences. If you need to dig into the internal
   implementation
   of SubArrays, you can find documentation on the changes for 0.4 here:
   http://docs.julialang.org/en/latest/devdocs/subarrays/
  
   OK, I'll have to study this in order to understand the implications for
 
  NamedArrays.
 
  ---david
 
   --Tim
  
   On Friday, December 05, 2014 04:18:08 AM David van Leeuwen wrote:
Hi,
   
On Friday, December 5, 2014 8:47:22 AM UTC+1, Ján Dolinský wrote:
 Hi,

 I am exploring DataFrames and NamedArrays packages and I would
 like to
  
   ask
  
 whether their are suitable for heavier computations and whether I
 can
  
   use
  
 them directly in BLAS calls (e.g. gemv() etc.). In addition, is it
 possible
 to create views of e.g. DataFrames or NamedArrays ?

 I can only speak for NamedArrays.  On the one hand the deployment
 of
  
   BLAS
  
should be transparant and the use of NamedArray vs Array not lead to
  
   much
  
degradation in performance.  E.g., a * b with `a` and `b` a
  
   NamedArray,
  
effectively calls a.array * b.array which Base implements with
BLAS.gemm().  There is just a little overhead of filling in sensible
  
   names
  
in the result---so if you have small matrices in an inner loop,
 you're
going to get hurt.
   
On the other hand, I am not sure how much of the Julia BLAS
 cleverness
  
   is
  
retained in NamedArrays---but the intention of the package is that
 it is
completely transparent, and if you notice bad performance for a
  
   particular
  
situation then you should file an issue (or make a PR:-).  Individual
element indexing of a NamedArray with integers is just a little bit
  
   slower
  
than that of an Array.  Indexing by name is quite a bit slower---you
 may
try a different Associative than the standard Dict.
   
Incidentally, I've been toying with the idea of NamedArrays `*`
 check on
consistency of index and dimension names, but my guess is that people
  
   would
  
find such a thing annoying.
   
ArrayViews are currently not aware of NameArrays.  I believe the
 views
  
   are
  
going to be part ov julia-0.4, so then it would be a task for
 NamedArray
  
   to
  
implement views of NamedArrays I 

Re: [julia-users] How fast reduce the Vector range of values? (Recode for smaler numbers)

[Tom Short]

You could look to see if the PooledDataArray in the DataArrays package will
do what you are after or at least give you ideas.
On Dec 4, 2014 12:35 PM, paul analyst paul.anal...@mail.com wrote:

 I have :
 Julia J
 1
 5
 1000
 2
 1000
 1


 i need recode to lowest values as posible
 Julia J
 1
 3
 4
 2
 4
 1

 Paul





 W dniu czwartek, 4 grudnia 2014 17:37:15 UTC+1 użytkownik Stefan Karpinski
 napisał:

 What are you trying to do? Reduce each value in J mod 1000?

 On Thu, Dec 4, 2014 at 10:44 AM, paul analyst paul.a...@mail.com wrote:

 I have a long vector J, which has little value. Unfortunately, these values
 are very different (small and large) J = [1 3 7 2 12548745 2
 125485458544363456345 125485458544363456345 ... 3 12,548,545,854].
 The following is a very slow like:
 julia J
 1846623-element Array{Int64,1}:
 930070
1475172
21474836496435345
21474836496435345
 4296445417
641
130
130
143
 ...
 4296444827
143
148
144
145

 J1=zeros(J)
 suJ=sort(unique(J))
 l=length(suJ)

 for i =1:l
 temp=findin(J,[suJ[i]])
 J1[temp]=i;if mod(i,1000).==0 println(i) end;
 end;

 How fast reduce the Vector range of values (Recode for smaler numbers)?
 Paul

Re: [julia-users] Dict performance with String keys

[Tom Short]

You could try using symbols instead of strings. Replace t with symbol(t).
On Nov 19, 2014 8:06 PM, Greg Lee egreg...@gmail.com wrote:

 Is there a faster way to do the following, which builds a dictionary of
 unique tokens and counts?

 function unigrams(fn::String)
 grams = Dict{String,Int32}()
 f = open(fn)
 for line in eachline(f)
 for t in split(line)
 i = get(grams,t,0)
 grams[t] = i+1
 end
 end
 close(f)
 return grams

 end


 On a file with 1.9M unique tokens, this is 8x slower than Python written
 in the same style.  The big hit comes from string keys; using int keys is
 closer to Python's performance.  Timings:  Julia 1083s, Python 126s, c++
 80s.

Re: [julia-users] introspection and captured variables

[Tom Short]

Nikolas, I haven't touched the code in a while, so it might not run
with Julia 0.3, but you might look here for some ideas:

https://github.com/tshort/Sims.jl

It's a way to create ODE's from components, including electrical
models. Models are custom expressions built up from symbolic objects.


On Tue, Oct 14, 2014 at 2:14 PM, Nikolas Tezak nikolas.te...@gmail.com wrote:
 Hi all,

 in my research I run numerical simulations (ODEs and SDEs) for circuit
 models that can be composed, i.e.,
 each system has an ode that modifies in-place the elements of an output
 array based on the current state variable.
 Moreover, dimensionality of each system may vary.

 function sys1_ode(t, x, xdot)
 xdot[1] = # some function of x, t
 xdot[2] = # some other function of x, t
 end


 function sys2_ode(t, x, xdot)
 xdot[1] = # some expression with x, t
 xdot[2] = # some other expression with x, t
 xdot[3] = # some other expression with x, t
 end


 What I would like to do is use metaprogramming to construct a combined ode
 for both systems where the state vectors are just stacked. For each system I
 compute the offset within the state vector (0 for sys1 and 2 for sys2) and
 then modify and recombine the function code as follows

 function sys12_ode(t, x, xdot)
 xdot[1+0] = # some expression with x[1:2], t
 xdot[2+0] = # some other expression with x[1:2], t
 xdot[1+2] = # some expression with x[1+2:3+2], t

 xdot[2+2] = # some expression with x[1+2:3+2], t
 xdot[3+2] = # some expression with x[1+2:3+2], t

 end


 So far, that would seem to be quite straightforward and I think I could get
 this working by calling code_lowered(sys1_ode) and using the rewritten
 output to construct an AST for the combined function.

 The difficulty now arises when my sys1 and sys2 odes are defined with some
 internal parameters that are not passed as an argument but rather through a
 closure from the surrounding scope, i.e. I have some ode factory:

 funcion make_sys1_ode(p1, p2)
 function sys1_ode(t, x, xdot)
 xdot[1] = # some expression with x, t AND p1, p2
 xdot[2] = # some expression with of x, t AND p1, p2
 end
 sys1_ode
 end

 Given the constructed sys1_ode method is there someway to dynamically access
 the captured variables, i.e. those that come from its closure?
 Otherwise, I suppose I could resort to passing parameters via an extra ODE
 argument, but it would be super nice if I could avoid this.
 The reason why I am trying to implement things this way is to speed up ODE
 evaluation for very large complex (i.e. nested) circuits by expanding out
 all ODE bodies into a single function.

 I hope this write-up makes sense.
 Thanks,
 Nik

Re: [julia-users] DataFrame groupby error

[Tom Short]

Frederico, the best way to solve this is to provide a reproducible
example. If you file a bug report at the following with an example,
there'll be less of a chance that this gets lost:

https://github.com/JuliaStats/DataFrames.jl/issues/

On Wed, Oct 8, 2014 at 3:32 PM, Frederico Novaes
frederico.nov...@gmail.com wrote:
 Hi,

 I have a DataFrame with 10 columns, say :c1 to :c10. I need to do a
 groupby, grouping by 4 of these columns. When I try,

 $ groupby(df,[:c1,:c3,:c4,:c7])

 I get the following error:

 MemoryError()
 while loading In[13], in expression starting on line 1

  in groupsort_indexer at
 /home/cloudopen/.julia/v0.3/DataArrays/src/grouping.jl:7
  in groupsort_indexer at
 /home/cloudopen/.julia/v0.3/DataArrays/src/grouping.jl:5
  in groupby at
 /home/cloudopen/.julia/v0.3/DataFrames/src/groupeddataframe/grouping.jl:44


 Any help ?

 Thanks.

Re: [julia-users] LLVM.js

[Tom Short]

With the new static compilation capabilities (thanks Jeff, Keno, and
Jameson!) and the new compile=all option, Julia can generate a large
LLVM bitcode file using the following (in the devel version of Julia):

cd julia/base
mkdir ../tmp
../julia --build /home/tshort/julia/tmp --dump-bitcode=yes
--compile=all -J /home/tshort/julia/usr/lib/julia/sys.ji -f sysimg.jl

After that, you can compile functions from the julia/tmp/sys.bc
bitcode file to JavaScript with something like (find the names of
functions in sys.bc with: llvm-nm sys.bc):

cd ../tmp
emcc -v sys.bc  -o out.js -s EXPORTED_FUNCTIONS=['_julia_abs;104547']

I've gotten individual functions like this to compile as well as pisum
from julia/test/perf/micro/perf.jl. In doing this, I've come across a
couple of items:

* The sys.bc needs to be a 32-bit build. I haven't managed that, yet.
The devel versions have been a bit goofed lately for 32-bit use.

   More info: https://code.google.com/p/nativeclient/issues/detail?id=3931

* The current Emscripten has a bug with some Julia-generated bitcode.

   More info: https://code.google.com/p/nativeclient/issues/detail?id=3932

I've managed to compile about 90% of libjulia using Emscripten. I had
to cut out most of the code related to libuv. Unfortunately, I haven't
gotten any code to compile that used libjulia. Although, I've gotten
90% of libjulia to compile, the missing 10% is called a lot. Still
more work to do there. My attempt involved hacking up the Makefiles. I
better attempt would involve making a new target to compile
libjulia.bc.

The bottom line is that I think this'll work someday, but it will take
some work.

Tom

On Thu, Oct 2, 2014 at 11:08 AM, JobJob jobbu@gmail.com wrote:
 Any updates on this?

 On Friday, 13 December 2013 15:16:31 UTC+2, tshort wrote:

 I've played a little with this. Using Jameson's static compile branch, I
 was able to dump some functions compiled by Julia to LLVM IR and compile
 these with Emscripten. I did have to mess with some symbol names because
 Emscripten doesn't like Julia's naming. See an Emscripten issue here:

 https://github.com/kripken/emscripten/issues/1888

 I also took a quick look at compiling openlibm, and I ran into some
 nonportable header stuff that would need to be worked on.

 The nice thing about trying to get compiled stuff to run is that you don't
 necessarily need all of Julia compiled. That means faster downloads, and
 that we don't have to get everything working at the beginning.

 It'd be great if we could position Julia to be the leading numerical
 language for the web. With both Firefox and Chrome running asm.js within 2 -
 4X of native, I think there's lots of opportunity here.



 On Fri, Dec 13, 2013 at 12:22 AM, John Myles White johnmyl...@gmail.com
 wrote:

 I think it would also be great to think a bit about how we might use
 Julia to generate LLVM IR to generate Javascript for certain simple web
 tasks. Writing Julia code and then letting a package compile it into an
 includable Javascript file could be really fun.

  — John

 On Dec 12, 2013, at 9:19 PM, Stefan Karpinski ste...@karpinski.org
 wrote:

  I’m not sure how practical it really is to wait until runtime to
  compile your code rather than precompiling it
 
  It's pretty frigging practical, as it turns out. This is great. More
  work in this direction and we may actually be able to run a full Julia
  instance in a browser.
 
 
  On Fri, Dec 13, 2013 at 12:14 AM, John Myles White
  johnmyl...@gmail.com wrote:
  The Emscripten folks are doing some really cool stuff:
  http://badassjs.com/post/39573969361/llvm-js-llvm-itself-compiled-to-javascript-via
 
   — John
 
 

Re: [julia-users] compilation of Julia code to shared libraries

[Tom Short]

As more background, see here:

https://groups.google.com/forum/#!topic/julia-dev/qdnggTuIp9s



On Wed, Jul 2, 2014 at 11:05 AM, Alexander Braumann
defender.bre...@gmail.com wrote:
 Hi,

 I am mainly using R and wanted to know if it was already possible (or when
 it is expected to be possible) to compile Julia code to shared libraries
 which can be loaded and used within R?

 I found a rather short discussion about this here:

 http://stackoverflow.com/questions/9965747/linking-r-and-julia

 The Julia development plan, as I described in this answer is to allow
 compilation of Julia code to shared libraries, callable using the C ABI.
 Once this happens, it will be as easy to call Julia code from R as it is to
 call C/C++ code. There is, however, a fair amount of work required before
 this becomes possible (Stefan Karpinski)

 but I couldn't comment there and therefore need to ask in this group.

 I am really looking forward to this feature! It would encourage me using
 julia a lot more.


 cheers,

 alex

Re: [julia-users] Reducing algorithm obfuscation

[Tom Short]

This obfuscation is also tedious with DataFrames.

I've been playing around with an `@with` macro to use symbols to
reference DataFrame columns. I extended that idea to several macros to
ease data manipulation:

https://github.com/JuliaStats/DataFramesMeta.jl

The tricky part with DataFrames and associative types is that the
types are not known ahead of time. Wrapping the code in the block in a
pseudoanonymous function gets around that issue. For DataFrames and
associative types, this also gives speed advantages because indexing
these is slow. I'm not sure if this approach could also cover mutable
and immutable objects.

Here is an example:

df = DataFrame(x = 1:3, y = [2, 1, 2])
x = [2, 1, 0]

@with(df, :y + 1)
@with(df, :x + x)  # the two x's are different

x = @with df begin
res = 0.0
for i in 1:length(:x)
res += :x[i] * :y[i]
end
res
end

On Fri, Jun 6, 2014 at 3:17 AM, Andrew Simper andrewsim...@gmail.com wrote:
 In implementations where you want named data, I've noticed that the
 algorithm gets obfuscated by lots of variable names with dots after them.
 For example, here is a basic analog model of a state variable filter used as
 a sine wave generator:

 immutable SvfSinOscCoef
 g0::Float64
 g1::Float64
 end
 immutable SvfSinOsc
 ic1eq::Float64
 ic2eq::Float64
 end
 function SvfSinOscCoef_Init (;freq=1.0, sr=44100.0)
 local g::Float64 = tan (2pi*freq/sr)
 local g0 = 1.0/(1.0+g^2)
 SvfSinOscCoef (g0,g*g0)
 end
 function SvfSinOsc_Init (startphase::Float64)
 SvfSinOsc (cos(startphase), sin(startphase))
 end

 But the tick function looks a bit messy:

 function tick (state::SvfSinOsc, coef::SvfSinOscCoef)
 local v1::Float64 = coef.g0*state.ic1eq - coef.g1*state.ic2eq
 local v2::Float64 = coef.g1*state.ic1eq + coef.g0*state.ic2eq
 SvfSinOsc (2*v1 - state.ic1eq, 2*v2 - state.ic2eq)
 end


 It would be really cool if there was a way to shorthand the syntax of this
 to something like the following, which is a lot more readable:

 function tick (state::SvfSinOsc, coef::SvfSinOscCoef)
 using s, c
 local v1::Float64 = g0*ic1eq - g1*ic2eq
 local v2::Float64 = g1*ic1eq + g0*ic2eq
 SvfSinOsc (2*v1 - ic1eq, 2*v2 - ic2eq)
 end


 Lots of algorithms have arguments with the same type, but even then you
 could still specify using just the most used argument, but if it doesn't
 help make things more clear or isn't useful then people don't have to use it
 at all.



 Another pattern that would be nice to handle cleanly is: fetch state to
 local, compute on local, store local to state. I have written code that
 generates code to handle this since it is such a pain to keep everything in
 sync, but if there was some way to automate this at the language level then
 it would really rock, so here is an example of the longhand way, which isn't
 too bad for this example, but just imagine if there are 20 or so variables,
 and you are writing multiple tick functions:

 type SvfSinOsc
 ic1eq::Float64
 ic2eq::Float64
 end

 function tick (state::SvfSinOsc, coef::SvfSinOscCoef)
 local ic1eq::Float64 = state.ic1eq
 local ic2eq::Float64 = state.ic2eq
 for i = 1:100
 # compute algorithm using local copies of state.ic1eq and
 state.ic2eq
 end
 state.ic1eq = ic1eq
 state.ic2eq = ic2eq
 return state
 end


 I have a feeling that macros may be able to help out here to result in
 something like:

 function tick (state::SvfSinOsc, coef::SvfSinOscCoef)
 @fetch state
 for i = 1:100
 # compute iterative algorithm using local copies of state.ic1eq and
 state.ic2eq
 end
 @store state
 return state
 end

 But I'm not sure how to code such a beast, I tried something like:

 macro fetch(obj::SvfSinOsc)
 return quote
 local ic1eq = obj.ic1eq
 local ic2eq = obj.ic2eq
 end
 end

 macro store(obj::SvfSinOsc)
 return quote
 obj.ic1eq = ic1eq
 obj.ic2eq = ic2eq
 end
 end

 dump(osc)
 macroexpand (:(@fetch osc))
 macroexpand (:(@store osc))

 SvfSinOsc
   ic1eq: Float64 1.0
   ic2eq: Float64 0.0


 Out[28]: :($(Expr(:error,
 TypeError(:anonymous,typeassert,SvfSinOsc,:osc

Re: [julia-users] variable binding with Expr

[Tom Short]

If you mean replace a by 1 in the expression, you can do it as follows:

julia e = :(typeof(a) : Real  typeof(b) : Real)
:(typeof(a) : Real  typeof(b) : Real)

julia dump(e, 10)# look at the structure of e
Expr
  head: Symbol 
  args: Array(Any,(2,))
1: Expr
  head: Symbol comparison
  args: Array(Any,(3,))
1: Expr
  head: Symbol call
  args: Array(Any,(2,))
1: Symbol typeof
2: Symbol a
  typ: Any
2: Symbol :
3: Symbol Real
  typ: Any
2: Expr
  head: Symbol comparison
  args: Array(Any,(3,))
1: Expr
  head: Symbol call
  args: Array(Any,(2,))
1: Symbol typeof
2: Symbol b
  typ: Any
2: Symbol :
3: Symbol Real
  typ: Any
  typ: Any

julia e.args[1].args[1].args[2]
:a

julia e.args[1].args[1].args[2] = 1
1

julia e
:(typeof(1) : Real  typeof(b) : Real)


On Tue, Jun 10, 2014 at 3:37 PM, Gustavo Lacerda
gus0...@optimizelife.com wrote:
 hi,

 julia parse(typeof(a):Realtypeof(b):Real)
 :(typeof(a) : Real  typeof(b) : Real)

 I would like to substitute 'a' for 1 in this Expr.  Any suggestions?

 Gustavo


 --
 Gustavo Lacerda
 http://www.optimizelife.com

Re: [julia-users] Yet another licensing question: Building a Julia version of an R package

[Tom Short]

Instead of asking the package authors to do extra work, it might be
better to offer to convert their package to a Julia package. Then, you
can ask the authors if the converted package could be released with an
MIT license.

On Mon, Mar 24, 2014 at 2:18 AM, Ted Fujimoto tftur...@gmail.com wrote:
 Thanks Jake! I'll also ask if they are willing to participate in the Julia
 community by implementing a Julia version too! :)


 On Sunday, March 23, 2014 6:14:33 PM UTC-7, Jake Bolewski wrote:

 Another strategy is to contact the authors directly and ask them if they
 would consider relicensing their work.  Many people do not really consider
 the implications of choosing one license over another and just go with a
 default.

 On Sunday, March 23, 2014 8:59:58 PM UTC-4, John Myles White wrote:

 Yes, including the same GPL-3 license is sufficient if you've derived
 your work from a GPL-3 project. You may also need to include the original
 headers of the files if they contain attribution information that you are
 required to preserve.

 I don't think there's anything dishonest about creating a GPL-3 package.
 If you would like to release something under a permissive license, you'll
 have to implement your code from scratch without ever reading any of the
 code from a GPL or closed-source implementation.

 What's most beneficial depends on context. Many businesses prohibit GPL
 software, so many people in the Julia (and Python) communities intentionally
 produce MIT or BSD software. But Julia benefits a lot from having GPL
 packages when there's no reasonable alternative.

  -- John

 On Mar 23, 2014, at 1:17 PM, Ted Fujimoto tftu...@gmail.com wrote:

 Hi all,

 I'm trying to familiarize myself with Julia by seeing how it compares to
 other languages. I would also like to open-source my code if it seems
 useful to others. Unfortunately, licenses have made this process
 complicated.

 A tangible example:

 I am trying to implement a Julia version of the R package pcalg
 (http://cran.r-project.org/web/packages/pcalg/index.html). Like most R
 packages, it is protected under the GPL-3 license. Also, the license states
 that it would consider my implementation a modification of the R package.
 Say I feel that my project is ready to be open-sourced and put it in a
 github repository. Is it enough to follow the RmathDist.jl lead and do the
 following?:
 1. Include the same license in the repository.
 2. Cite the R package I modified.

 A more long term question: I'm guessing a better (and more honest)
 alternative to the above would be to implement the relevant algorithms by
 looking at the pseudocode and applying it in a way that is friendlier to
 future improvements using idiomatic Julia (if it exists yet). After that,
 open-source it under the MIT license. Would this be a more beneficial
 approach than the Julia version of an R package approach?

Re: [julia-users] C++-like operator() for Julia's composite types

[Tom Short]

Does this give you what you want:

function make_interp(x, y)
y = copy(y)
function interp(x2)
@assert x2  x[1]  x2  x[end]
i = 1
while x[i]  x2; i += 1; end
(x[i] - x2) * y[i-1] + (x2 - x[i-1]) * y[i]
end
end

x = [1,2,3,4,5]
y = [x2^2 for x2 in x]

my_interp = make_interp(x, y)
println(my_interp(2.1))  # prints 4.5
y[2] = 5
println(my_interp(2.1))  # prints 4.5


On Tue, Feb 25, 2014 at 4:01 PM, Marek Gagolewski
m.gagolew...@gmail.com wrote:
 Dear Andrew,

 Nope, unfortunately it's not what I am trying to achieve. The code you've
 kindly (let's forget the interpolation task, what I really meant is some
 programming construct, so it's OK with no delta_x) submitted gives:

 my_interp = make_interp(x, y)
 println(my_interp(2.1))  # prints 4.5

 y[2] = 5
 println(my_interp(2.1))  # prints 5.4

 and I also would like to get 4.5 in the second case.

 All the best,
 Marek

Re: [julia-users] Re: Error: no method display(DataFrame)

[Tom Short]

That works, but columns will be Arrays instead of DataArrays. That's
the way it's always worked. If you want them to be DataArrays, then
convert to DataArrays right at the end.

To fix show to support columns that are arrays, we probably need (at
least) to define the following:

countna(da::Array) = 0



On Thu, Jan 23, 2014 at 4:07 PM, Jacob Quinn quinn.jac...@gmail.com wrote:
 Great investigative work. Is
 DataFrames( array_of_arrays, Index(column_names_array) )
 not the right way to hand construct DataFrames any more? I think I can
 allocate DataArrays instead, but at every step of the way, I was trying to
 hand-optimize the result fetching process, which resulted in not creating a
 DataArray or DataFrame until right before we return to the user.

 -Jacob


 On Thu, Jan 23, 2014 at 3:27 PM, bp2012 bert.pritch...@gmail.com wrote:

 To check Jacob's suggestion about versions mismatch I completely removed
 the DataFrames and ODBC packages using Pkg.rm and physically deleted the
 directories from disk. I then added them via Pkg.add and Pkg,update.

 I am running the julia nightlies build.
 julia versioninfo()
 Julia Version 0.3.0-prerelease+1127
 Commit bc73674* (2014-01-22 20:09 UTC)

 Pkg.status()
  - DataFrames  0.5.1
  - ODBC  0.3.5

 Pkg.checkout(ODBC)
 INFO: Checking out ODBC master...
 INFO: Pulling ODBC latest master...
 INFO: No packages to install, update or remove

 julia Pkg.checkout(DataFrames)
 INFO: Checking out DataFrames master...
 INFO: Pulling DataFrames latest master...
 INFO: No packages to install, update or remove

 I did some digging. It looks like there is a mismatch in that countna
 expects DataFrame columns to be DataArrays. However the ODBC package returns
 DataFrames that have array columns (using the first constructor in
 dataframe.jl). You guys would know better as to whether a change is needed
 in the constructor or if countna should also accept Array columns.


 I made some local changes to work around the issue.

 show.jl:
 line 42:  if isna(col, i) changed to  if isna(col[i])
 line 322:  missing[j] = countna(adf[j]) changed tomissing[j] =
 countna(isa(adf[j], DataArray) ? adf[j] : DataArray(adf[j]))

 These work great for me.

Re: [julia-users] Re: Error: no method display(DataFrame)

[Tom Short]

I think of item #3 as a feature, not a bug. I don't like the idea of
auto-conversion. If I choose Vectors, I should not expect them to
support missing values. R sometimes irritates me by adding NA's when I
don't expect it. I'd rather have the error than have NA's sneak in
there. Also, there may be other types of AbstractDataFrames where we
don't have the ability to assign missing values. HDF5 tables are one
example I can think of. We wouldn't want to try to autoconvert a huge
HDF5 column to a DataVector.



On Thu, Jan 23, 2014 at 8:58 PM, John Myles White
johnmyleswh...@gmail.com wrote:
 A couple of points that expand on Tom’s comments:

 (1) We need to add Tom’s definition of countna(a::Array) = 0 to show() wide 
 DataFrame’s that contain any columns that are Vector’s. I never use 
 DataFrame’s like that, so I forgot that others might. It’s also impossible to 
 produce such a DataFrame using our current I/O routines.

 (2) The constructor you’re using does exist, Jacob, but you should typically 
 pass in a Vector{Any}, each element of which is either a DataVector or 
 PooledDataVector. See Point (3) for why, at the moment, using a Vector as a 
 column is subtly broken.

 (3) If people are going to put Vector’s in DataFrames for performance 
 reasons, all of our setindex!() functions for DataFrames need to add methods 
 that automatically convert Vector’s to DataVector’s if an NA is inserted in a 
 Vector. Right now that kind of insertion is just going to error out. Ths 
 check isn’t too hard, but it’s totally missing from our current codebase.

 Personally, I would prefer that we not allow any of the columns of a 
 DataFrame to be Vector's. It’s a weird edge case that doesn’t actually offer 
 reliable high performance, because the potential performance improvements 
 relies on the unsafe assumption that a DataFame won’t contain any columns 
 with NA’s in it.

  — John

 On Jan 23, 2014, at 1:33 PM, Tom Short tshort.rli...@gmail.com wrote:

 That works, but columns will be Arrays instead of DataArrays. That's
 the way it's always worked. If you want them to be DataArrays, then
 convert to DataArrays right at the end.

 To fix show to support columns that are arrays, we probably need (at
 least) to define the following:

 countna(da::Array) = 0



 On Thu, Jan 23, 2014 at 4:07 PM, Jacob Quinn quinn.jac...@gmail.com wrote:
 Great investigative work. Is
 DataFrames( array_of_arrays, Index(column_names_array) )
 not the right way to hand construct DataFrames any more? I think I can
 allocate DataArrays instead, but at every step of the way, I was trying to
 hand-optimize the result fetching process, which resulted in not creating a
 DataArray or DataFrame until right before we return to the user.

 -Jacob


 On Thu, Jan 23, 2014 at 3:27 PM, bp2012 bert.pritch...@gmail.com wrote:

 To check Jacob's suggestion about versions mismatch I completely removed
 the DataFrames and ODBC packages using Pkg.rm and physically deleted the
 directories from disk. I then added them via Pkg.add and Pkg,update.

 I am running the julia nightlies build.
 julia versioninfo()
 Julia Version 0.3.0-prerelease+1127
 Commit bc73674* (2014-01-22 20:09 UTC)

 Pkg.status()
 - DataFrames  0.5.1
 - ODBC  0.3.5

 Pkg.checkout(ODBC)
 INFO: Checking out ODBC master...
 INFO: Pulling ODBC latest master...
 INFO: No packages to install, update or remove

 julia Pkg.checkout(DataFrames)
 INFO: Checking out DataFrames master...
 INFO: Pulling DataFrames latest master...
 INFO: No packages to install, update or remove

 I did some digging. It looks like there is a mismatch in that countna
 expects DataFrame columns to be DataArrays. However the ODBC package 
 returns
 DataFrames that have array columns (using the first constructor in
 dataframe.jl). You guys would know better as to whether a change is needed
 in the constructor or if countna should also accept Array columns.


 I made some local changes to work around the issue.

 show.jl:
 line 42:  if isna(col, i) changed to  if isna(col[i])
 line 322:  missing[j] = countna(adf[j]) changed tomissing[j] =
 countna(isa(adf[j], DataArray) ? adf[j] : DataArray(adf[j]))

 These work great for me.

Re: [julia-users] New Year's resolutions for DataArrays, DataFrames and other packages

[Tom Short]

I also agree with your approach, John. Based on your criteria, here
are some other things to consider for the chopping block.

- expression-based indexing
- NamedArray (you already have an issue on this)
- with, within, based_on and variants
- @transform, @DataFrame
- select, filter
- DataStream

Many of these were attempts to ease syntax via delayed evaluation. We
can either do without or try to implement something like LINQ.



On Mon, Jan 20, 2014 at 7:02 PM, Kevin Squire kevin.squ...@gmail.com wrote:
 Hi John,

 I agree with pretty much everything you have written here, and really
 appreciate that you've taken the lead in cleaning things up and getting us
 on track.

 Cheers!
Kevin


 On Mon, Jan 20, 2014 at 1:57 PM, John Myles White johnmyleswh...@gmail.com
 wrote:

 As I said in another thread recently, I am currently the lead maintainer
 of more packages than I can keep up with. I think it’s been useful for me to
 start so many different projects, but I can’t keep maintaining most of my
 packages given my current work schedule.

 Without Simon Kornblith, Kevin Squire, Sean Garborg and several others
 doing amazing work to keep DataArrays and DataFrames going, much of our
 basic data infrastructure would have already become completely unusable. But
 even with the great work that’s been done on those package recently, there’s
 still lot of additional design work required. I’d like to free up some of my
 time to do that work.

 To keep things moving forward, I’d like to propose a couple of radical New
 Year’s resolutions for the packages I work on.

 (1) We need to stop adding functionality and focus entirely on improving
 the quality and documentation of our existing functionality. We have way too
 much prototype code in DataFrames that I can’t keep up with. I’m about to
 make a pull request for DataFrames that will remove everything related to
 column groupings, database-style indexing and Blocks.jl support. I
 absolutely want to see us push all of those ideas forward in the future, but
 they need to happen in unmerged forks or separate packages until we have the
 resources needed to support them. Right now, they make an overwhelming
 maintenance challenge even more onerous.

 (2) We can’t support anything other than the master branch of most
 JuliaStats packages except possibly for Distributions. I personally don’t
 have the time to simultaneously keep stuff working with Julia 0.2 and Julia
 0.3. Moreover, many of our basic packages aren’t mature enough to justify
 supporting older versions. We should do a better job of supporting our
 master releases and not invest precious time trying to support older
 releases.

 (3) We need to make more of DataArrays and DataFrames reflect the Julian
 worldview. Lots of our code uses an interface that is incongruous with the
 interfaces found in Base. Even worse, a large chunk of code has
 type-stability problems that makes it very slow, when comparable code that
 uses normal Arrays is 100x faster. We need to develop new idioms and new
 strategies for making code that interacts with type-destabilizing NA’s
 faster. More generally, we need to make DataArrays and DataFrames fit in
 better with Julia when Julia and R disagree. Following R’s lead has often
 lead us astray because R doesn’t share Julia’s strenths or weaknesses.

 (4) Going forward, there should be exactly one way to do most things. The
 worst part of our current codebase is that there are multiple ways to
 express the same computation, but (a) some of them are unusably slow and (b)
 some of them don’t ever get tested or maintained properly. This is closely
 linked to the excess proliferation of functionality described in Resolution
 1 above. We need to start removing stuff from our packages and making the
 parts we keep both reliable and fast.

 I think we can push DataArrays and DataFrames to 1.0 status by the end of
 this year. But I think we need to adopt a new approach if we’re going to get
 there. Lots of stuff needs to get deprecated and what remains needs a lot
 more testing, benchmarking and documentation.

  — John

Re: [julia-users] Declaring types that are special cases of already existing composite types

[Tom Short]

There are other mailing list threads on this. See here for one:

https://groups.google.com/d/msg/julia-users/yCjsY9g9_Lg/4oYlkf3DFHYJ

For your Portfolio use case, I'd be tempted to stick with DataFrames
and usage conventions. Maybe add a `checkportfolio` function to ensure
consistency. Another option is a thin wrapper type that includes an
AbstractDataFrame. If you don't want to write the getindex and
setindex! functions, then maybe just functions to convert between
Portfolios and AbstractDataFrames would be enough.



On Thu, Jan 2, 2014 at 9:23 AM, Christian Groll
groll.christian@gmail.com wrote:
 My interest lies in the implementation of types that are special cases of
 already existing composite types.

 For example, I want to implement a type Portfolio, which is just a DataFrame
 with two additional requirements:
 - all columns are of numeric type
 - the sum of the entries in each row must be equal to 1

 Or, I want to implement a type TimeSeries, which is just a DataFrame where
 the first column consists of dates.

 I think the way to go here would be to implement some type of constraint
 checking in the setindex! methods, although this would not prevent messing
 with the entries by way of directly setting the fields of the type. However,
 once there exist convenient getindex and setindex! methods, I hope that
 basically nobody would mess with the values directly, and complete
 immutability is not necessarily needed.

 What I am trying to achieve is something that I think is called inheritance
 in other languages, where classes can simply be declared as subclasses to
 already existing classes.

 So the question is, whether something like this is possible in julia as
 well? As far as I get it, there is no way to declare a type to be a subtype
 of a composite type.
 Or, if this is not possible, is there any way around, like for example
 declaring a type portfolio,

 type Portfolio
weights::DataVector{Float64}
 end

 where I can simply relate all setindex! methods to the respective methods of
 DataVector, adding constraint checks where necessary.

 function Base.setindex!(pf::Portfolio,
 v::Any,
 col_ind::ColumnIndex)
 if check_constraints(pf, v, col_ind)
setindex!(pf.weights, v, col_ind)
 else
error(constraints not fulfilled.)
 end
 end

 However, I then would need some type of metaprogramming such that I do not
 have to implement all the numerous setindex! methods of DataVector from
 scratch up.