File.ls doesn’t appear to work with wildcards, I get this error `{:error,
:enoent}`. But File.ls(“lib”) is also fast. The C script and NIF are beyond
my knowledge.
On Monday, October 14, 2019 at 12:35:23 PM UTC-6, José Valim wrote:
>
> Right. Now try running File.ls in some directories and see if that is
> enough to reproduce the issue. Then I would implement a C script that calls
> ls as a NIF, see if it is fast or not. And so on.
>
>
> *José Valim*
> www.plataformatec.com.br
> Skype: jv.ptec
> Founder and Director of R&D
>
>
> On Mon, Oct 14, 2019 at 8:29 PM Michael St Clair <micha...@gmail.com
> <javascript:>> wrote:
>
>> What is the best way to experiment with that? As I am showing that using
>> the ls command is much faster (times below)
>>
>> On my Mac:
>>
>> iex(3)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0)
>> |> Kernel./(1_000_000)
>>
>> 0.095601
>>
>> iex(4)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0)
>> |> Kernel./(1_000_000)
>>
>> 0.071896
>>
>>
>> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |>
>> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |>
>> Kernel./(1_000_000)
>>
>> 0.034176
>>
>> iex(6)> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |>
>> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |>
>> Kernel./(1_000_000)
>>
>> 0.028481
>>
>>
>> On our production instance running docker (elixir:1.9.1-alpine):
>>
>> iex(zipbooks@phoenix-360h)1> fn -> Path.wildcard("lib/**/*.ex") end |> :
>> timer.tc |> elem(0) |> Kernel./(1_000_000)
>>
>> 0.448345
>>
>> iex(zipbooks@phoenix-360h)2> fn -> Path.wildcard("lib/**/*.ex") end |> :
>> timer.tc |> elem(0) |> Kernel./(1_000_000)
>>
>> 0.35429
>>
>>
>> iex(zipbooks@phoenix-360h)7> fn -> "ls lib/**/*.ex" |>
>> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end
>> |> :timer.tc |> elem(0) |> Kernel./(1_000_000)
>>
>> 0.00361
>>
>> iex(zipbooks@phoenix-360h)8> fn -> "ls lib/**/*.ex" |>
>> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end
>> |> :timer.tc |> elem(0) |> Kernel./(1_000_000)
>>
>> 0.003731
>>
>>
>> Local docker container with mounted directories(elixir:1.9.1-alpine):
>>
>> iex(1)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0)
>> |> Kernel./(1_000_000)
>>
>> 0.881692
>>
>> iex(2)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0)
>> |> Kernel./(1_000_000)
>>
>> 0.897629
>>
>>
>> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |>
>> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |>
>> Kernel./(1_000_000)
>>
>> 0.031227
>>
>> iex(4)> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |>
>> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |>
>> Kernel./(1_000_000)
>>
>> 0.031241
>>
>> On Monday, October 14, 2019 at 12:14:57 PM UTC-6, José Valim wrote:
>>>
>>> There is no way to patch out Mix.Utils to run your own code. The proper
>>> fix would be to figure out why the path traversal is slow on Elixir (it is
>>> implemented on top of filelib:wildcard) so we can fix it rather
>>> permanently. How long does "Path.wildcard("lib/**/*.ex")" take? Can you
>>> implement your own path traversal in Elixir? Is it faster than
>>> Path.wildcard?
>>>
>>>
>>> *José Valim*
>>> www.plataformatec.com.br
>>> Skype: jv.ptec
>>> Founder and Director of R&D
>>>
>>>
>>> On Mon, Oct 14, 2019 at 8:08 PM Michael St Clair <micha...@gmail.com>
>>> wrote:
>>>
>>>> I've been doing some digging around to see how to make phoenix run
>>>> faster in Docker. Without this plug the app runs just as fast as native
>>>> `plug(Phoenix.CodeReloader`. I think I have traced it down to this
>>>> https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/utils.ex#L237.
>>>>
>>>> Running just that function explains most if not all of the extra load time
>>>> on my endpoints. However this code `"ls lib/**/*.ex" |>
>>>> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") |>
>>>> IO.inspect` appears to do the same thing and much faster. I'm just running
>>>> into the problem of how to override that Mix.Utils file to test it here
>>>> https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/compilers/elixir.ex#L36.
>>>>
>>>> Any suggestions?
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elixir-lang-core" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to elixir-l...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/elixir-lang-core/b811d808-04a2-427e-9281-3d337f05de16%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/elixir-lang-core/b811d808-04a2-427e-9281-3d337f05de16%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "elixir-lang-core" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elixir-l...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elixir-lang-core/b46f9a3c-4f0a-479d-98df-a921ffeb65fc%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/elixir-lang-core/b46f9a3c-4f0a-479d-98df-a921ffeb65fc%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elixir-lang-core+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elixir-lang-core/13ce15ce-7bad-468a-84c5-10be4044935c%40googlegroups.com.
