On Tue, Mar 6, 2018 at 5:40 AM, Carsten Haitzler <ras...@rasterman.com> wrote:
> On Mon, 5 Mar 2018 11:00:27 -0300 Gustavo Sverzut Barbieri
> <barbi...@gmail.com>
> said:
>
>> On Sat, Mar 3, 2018 at 2:02 AM, Carsten Haitzler <ras...@rasterman.com>
>> wrote:
>> > I just pushed:
>> >
>> > eb0b826776b60e0d97218242a5c285d146fb6f3b
>> >
>> > https://git.enlightenment.org/core/efl.git/commit/?id=1bdd9e4dd15fc27da43b50fd29bfb1b0b30ef6bd
>> >
>> > I wrote up a high level design document and description here for an idea of
>> > how it works:
>> >
>> > https://phab.enlightenment.org/w/efl-loops-threads/
>> >
>> > I'm busy filling that document out a bit, but the core essentials are
>> > there.
>> >
>> > There are some details like loop having an exit eina value vs task having a
>> > simple int exit code (i'm sticking to this because it is what processes do,
>> > it's simple and its universally supported between processes, unlike an eina
>> > value). yes - it simplifies threads to only having int exit codes, but the
>> > simplicity of the design is what I'm going for. threads like executeables
>> > have full bytestream I/O for more complex data interchange. So there
>> > probably needs to be a bit of adjusting here and there to remove
>> > duplication. The Arguments event delivers arguments in an array, but the
>> > task object also stores arguments too permanently. Do we need to double-up
>> > the information? So a few small things like this.
>> >
>> > I've given this design a lot of thought and what I have here I think is
>> > clean and neat, tidy and fairly simple. It actually does work. I have
>> > tested it of course. But please have a look and let me know what you think.
>> > Are there some major defects in the design and idea? I know we can expand
>> > this in future with more controls (I have no pause/resume controls in the
>> > task interface but there should be. For processes use SIGSTOP/CONT and for
>> > threads a co-operative request on the control line). The internals could be
>> > better. I use pipes and this eats up a lot of FD's for the threads where I
>> > could use socketpairs instead. I have a separate control pipe in/out from
>> > I/O in/out where i could multiplex on the same socketpair. Currently we run
>> > out of FD's (after about 240 threads running at once because they eat up 8
>> > FD's per thread. The Efl.Io code across the classes is a lot of copy &
>> > paste... And I don't have anything to change priority of a thread once it
>> > has run (no eina API for this - needs to be added).
>> >
>> > I'm not that happy though with Efl.Io and the sheer amount of code needed
>> > to
>> > deal with it. Even as a user of the API.
>> >
>> > Anyway - comments, thoughts, etc. etc. ?
>>
>> Well, I already mentioned this to you in irc, but replying here just
>> to make my point:
>>
>> I think the design is upside down, trying to make life easier at some
>> point resulted in messy at the other side.
>
> how? where? how is it messy?
>
>> okay, call it a task, work for both process and threads, the hope is
>> to facilitate "switch from threads to process, and vice versa", but
>> we're getting the worst part of each it seems.
>>
>> ie: most thread usage is about sending shared memory back and forth,
>> after all the benefit of threads is just that: shared memory. What we
>> got? argv + int, useless for real life threads.
>
> i did this because not only does this work between processes it can work for
> js and lua. lua can't share objects between threads. the best youcan do is a
> luastate per thread and that pretty much means treating threads as another
> process. the model of at the base terating threads like processes is more
> widely applicable than "pass share objects" because several languages/runtimes
> just can't do it at all.
ok, for these cases, like for JS/WebWorkers the solution is to start a
new thread passing the "script to run in that thread", in the case of
JS/WebWorkers, there are message passing.
That said, to fully implement something like that you better use a
pointer (ie: string + mutex + queue), not a serialized value. Not a
stdio-like as it was done.
>> solution? "stdio" for threads... you can send/receive I/O, done with
>> pipes, like it would be for processes.
>
> it's the same thing ecore_thread does but only one way (from thread to main
> loop) and only in a limited (only main loop can spawn threads".
>
> so this is a far more expanded model with data going the other way too.
>
>> I really, really don't like that design. To me threads should be
>> exposed as such, with pointer argument and result... if you wish to
>> make it easier, you could "box" it someway, like eina_value... or just
>
> i did the above because it's going to universally work with things like lua
> etc.. there needs to be some kind of hook to create a new luastate in the
> thread before executing, but then it'd work as-is.
no, see above... you need *MORE*, unless you opt to serialize stuff as
argv -- which is bad -- or block the thread reading from "stdin" to
receive something -- which is also bad.
If you get to implement this, you'd -- on the main thread -- create a
string (ie: script contents to run, or pre-parsed AST) + mutex +
queues (or some "async queue" primitive we should provide). Then on
the second thread you setup the new VM with the script contents and
hooks that to the message passing queue, likely you need to inject in
the VM a new function that calls back your thread, that will
post/consume events from the queues.
You're trying to make it easier by providing "queue as stdio", so you
get some pre-canned notification + buffering using the kernel +
select/poll... Not sure it would be the best for Lua (I'd rather
notify AND then use an in-memory queue, with lower costs than
executing "read/write" to the kernel).
For Python and other languages that can work with threads, you'd just
ignore this stdio approach.
>> let the traditional "void *pointer". But that's fundamental, say for
>> Python bindings to use threads, you want to send PyObject pointer to
>> the thread, that object will contain the information to callback the
>> user (the user's Python-callable). And for most usages as well, like
>> sending file info to be opened/decoded, etc.
>
> i can easily add a "set a void ptr on the thread" and make it gettable from
> the
> appthread. it wont work with lua though. it can't work with exe's. it's kind
> of
> not that useful for python etc. because it's just a void ptr.
Raster, maybe you should go play with bindings for a bit. None of this
makes much sense... when you do binding you don't blindly expose every
single call... as some calls are meant to support your binding, not to
be exposed.
And "it can't work with exes" because it's not an interface to be
exposed to EXE, as it's a thread thing. Read my email again, we
shouldn't mix those, because they are different: for threads you DO
EXPECT shared memory... that's the whole point of threads. The
constructor should be different.
>> thread I/O, while useful in some cases, I don't see why they should
>> always exist. They should be optional, and as such they should be
>> created explicitly.
>
> the control pipes do need to exist to handle joining the threads - knowing
> when
> they exit (and to request) but the stdio i can make optional with flags like i
> did with exe's. i might move those flags to the task class out of the exe
> class
> then. i don't think explicit creation is necessary or even good. some flags
> should do the trick just fine.
I think that the pipe structure should be exposed as an Efl.Io own its
own... then you create one if needed. Sending its read-side to one
thread, the write side to another. It may be even useful to provide
pipes to non-EFL entities, like external libraries (you'd send the
pipe's fds but not use their "other end" wrapper objects)
I know there is an "eoid" thread access problem... that "protection"
is showing to be more and more flawed as we start trying to do
something useful. Not sure how you're going to solve that for
"worker-thread-access-app" (app being in the main thread) and the
likes, but needs to be done in a clean way, or just remove that eiod
protection that is just causing harm.
>> Last but not least, I'd expose the stack just like in the OS, to not
>> make confusion:
>>
>> - Application ("the binary") -> Processes -> Threads -> main loop
>>
>> you can present "getters" that make it easier to access (ie: main loop
>> "get" application, in addition to "get parent" which would return the
>> thread).
>
> i already mentioned this. we can't. app object would have to be a shared
> object. it then CANT have a non-shared child. it's a logical problem that a
> locked object can't then modify non-locked objects (parents modify children).
>
> but parents already have handles to child threads and can control them
> (co-operatively though efl takes care of being co-operative).
see, as I said above this "protection" is proving to be useless...
just harm at no benefit.
at first it looked a promising way to avoid mistakes, after some
experiments it's proving to be just a PITA.
>> But mixing stuff "in the hope to make it easier" it not, it's just
>> making things more complicated... ALSO note that the developer that
>> would use this kind of API is "not the average developer", these don't
>> mess with such low level primitives. The developer that is likely to
>> use these primitives will say "what the fuck is this? I'm used to
>> processes and threads, these guys just messed it up".
>
> i disagree. i've seen the use of threads pretty much because "your annual
> bonus depends on you using more threads" and then madness that ends up as a
> result.
well, world is not just samsung :-D
> making threads easier to access and communicate with and deal with is
> an important thing. hell - i want to make it easy for me too.
>
> ecore_thread is kind of horrible IMHO because of its major limitations in this
> regard. this fixes many of those aspects.
You're fixing it wrongly... as I said, thinking about efficiency this
"channel through kernel" is horrible. And we'd be compared against the
other technologies, all that provide some "async queue", which is
in-memory.
>> I'm in favor of interfaces for things that are the same, so if the
>> methods in process and threads share the same concept, behavior and
>> parameters, make them an interface... when switching from process x
>> threads one doesn't need to "sed" everything. However, definitely
>> constructors are NOT the same concept, behavior or parameters, thus
>> not part of the interface.
>
> to me threads and processes are the same with only a few minor differences:
>
> 1. a crash in a thread brings down everything, but in a process only brings
> itself down
> 2. threads by default share memory space (but accessing shared memory needs to
> be carefully controlled)
> 3. threads can't be sensibly killed off without side-effects for everyone else
this last point is not true, see the socks usage... you can declare it
as cancellable and the SYSCALLS will fail, it's not going to abort at
some random code path.
so it can be controlled and "sensibly killed" without side-effects.
> i can easily add a void * input and return in addition to args/exit code. i
> like the args/exit code and environment being universal because if you stick
> to
> this subset then things are portable between languages, runtimes and between
> threads and processes. i can add flags to explicitly enable i/o.
why do that instead of simply create your own I/O and pass it as the
void* that exist for the common thread case?
> what i haven't figured out yet or done is things like
> ecore_thread_async_call()
> (and a sync version) which of course isn't going to work for lua, but
> technically that's what the command pipe is for. i'll have to make it
> non-blocking though to use more than it is now just for exit and exited
> commands.
>
> so i'll add void ptrs for in/out in the thread class.
>
> i'll move i/o flags to task class from exe and use them to determine i/o
>
> then this covers your points i believe. in addition it still does args and so
> on like exe's.
why not just give it a try to isolate the pipe under efl.io.pipe and
cleanup the thread part to be app "void*" based? Then you can pass
it...
later you can even provide an "async queue" object, an in-memory queue
with proper locking and notification?
--
Gustavo Sverzut Barbieri
--------------------------------------
Mobile: +55 (16) 99354-9890
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
