Re: [Factor-talk] Hooking the Back End

[Björn Lindqvist]

In that scenario, you are supposed to start a new thread in which the
copying proceeds. Factor's IO is implemented as a blocking facade
around an asynchronous core. That's why the CopyFile call on Windows
is likely a bug, it is blocking for real. See this thread:
https://sourceforge.net/p/factor/mailman/message/32588643/

I also wonder if you are not working on a to low level for the
feature. The io backend hooks are mostly for encapsulating platform
differences and are hard to extend.


2016-06-15 0:18 GMT+02:00 Alexander Ilin :
> But there is still a point in having a separate batch-processing backend.
> You see, if you start a copy operation in the Listener interactively, then
> even with the good variant of the copy-file you won't be able to continue
> working with the Listener until the copying is finished. Sure, it won't
> hang, and the caret will keep blinking, but you won't be able to queue the
> next file to be copied while the first one is still being handled. That's
> what my custom backend would do for you.


--
mvh/best regards Björn Lindqvist

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421=/41014381
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Hooking the Back End

[Alexander Ilin]

15.06.2016, 00:20, "Doug Coleman" :Methods on object work for any object, even ``f``, but any more specific type will override and get called instead. Methods on object are default methods, but the windows-specific method is worse since it blocks.  I see, that's good to know, thank you. Is there a native windows api call that doesn't block, or should we just use the default copy-file? There is no asynchronous CopyFile, only ReadFileEx and WriteFileEx, which, I think, the default copy-file amounts to (or maybe it's ReadFile/WriteFile with the OVERLAPPED parameter - they are effectively the same as the *Ex). That's what the IO completion ports are used for. Knowing that, there is no point in having the copy-file version that calls the (synchronous) CopyFile. But there is still a point in having a separate batch-processing backend. You see, if you start a copy operation in the Listener interactively, then even with the good variant of the copy-file you won't be able to continue working with the Listener until the copying is finished. Sure, it won't hang, and the caret will keep blinking, but you won't be able to queue the next file to be copied while the first one is still being handled. That's what my custom backend would do for you. Another thing I'd like it to be able to do is to persist the pending operations and stop and resume them as needed across Factor runs. That's for the cases where I'd plan a file reorganization without actual access to the storage medium. Or an automated backup. Yes, I have some big plans in this area. So, as an experiment, I have separated some of the operations into a new dmc-backend (delete-move-copy) alongside the (lower-level) io-backend. This sort of works, although it can't be local to my vocab, as it requires an interface change in the Factor's basis. Having the separate dmc-backend hooking point allows me to (temporarily) substitute the "windows" singleton with my own "batch" singleton (using with-variable), and therefore use all the higher level words that call copy-file internally, such as copy-file-into or copy-files-into, etc. ---=---Александр --
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421=/41014381___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Hooking the Back End

[Doug Coleman]

Methods on object work for any object, even ``f``, but any more specific
type will override and get called instead. Methods on object are default
methods, but the windows-specific method is worse since it blocks.

Is there a native windows api call that doesn't block, or should we just
use the default copy-file?


On Tue, Jun 14, 2016 at 2:11 PM, Alexander Ilin  wrote:

> Hello, Doug!
>
> I'm a bit confused, because I found this code:
>
> IN: io.directories.windows
>
> M: windows copy-file ( from to -- )
> dup parent-directory make-directories
> [ normalize-path ] bi@ 0 CopyFile win32-error=0/f ;
>
> This code defines copy-file as a method on "windows", and the code you
> quoted below defines copy-file as a method on "object".
>
> In what circumstances io-backend will have an instance of an "object"
> (instead of windows, unix, etc.) so that the code you quoted is actually
> executed?
>
> Is it the default implementation in case "windows" has no "copy-file"
> method?
>
> 14.06.2016, 22:37, "Doug Coleman" :
>
> Hey Alexander,
>
> We have a copy-file implemented with stream-copy already. It's
> non-blocking and it might be fast enough, or we could optimize it. I
> haven't looked at CopyFile for a long time, but maybe we shouldn't use that
> and should use the default stream-copy version instead. Then you don't have
> to replace the io-backend with a different one.
>
> Doug
>
>
> ! From basis/io/directories/directories.factor
> IN: io.directories
>
> ! Copying files
> HOOK: copy-file io-backend ( from to -- ) ;
>
> M: object copy-file
> dup parent-directory make-directories
> binary  [
> swap binary  [
> swap stream-copy
> ] with-disposal
> ] with-disposal ;
>
>
> ---=---
> Александр
>
>
>
> --
> What NetFlow Analyzer can do for you? Monitors network bandwidth and
> traffic
> patterns at an interface-level. Reveals which users, apps, and protocols
> are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity
> planning
> reports.
> http://pubads.g.doubleclick.net/gampad/clk?id=1444514421=/41014381
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk
>
>
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421=/41014381___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Hooking the Back End

[Alexander Ilin]

Hello, Doug! I'm a bit confused, because I found this code: IN: io.directories.windows M: windows copy-file ( from to -- )    dup parent-directory make-directories    [ normalize-path ] bi@ 0 CopyFile win32-error=0/f ; This code defines copy-file as a method on "windows", and the code you quoted below defines copy-file as a method on "object". In what circumstances io-backend will have an instance of an "object" (instead of windows, unix, etc.) so that the code you quoted is actually executed? Is it the default implementation in case "windows" has no "copy-file" method? 14.06.2016, 22:37, "Doug Coleman" :Hey Alexander, We have a copy-file implemented with stream-copy already. It's non-blocking and it might be fast enough, or we could optimize it. I haven't looked at CopyFile for a long time, but maybe we shouldn't use that and should use the default stream-copy version instead. Then you don't have to replace the io-backend with a different one. Doug  ! From basis/io/directories/directories.factorIN: io.directories ! Copying filesHOOK: copy-file io-backend ( from to -- ) ; M: object copy-file    dup parent-directory make-directories    binary  [        swap binary  [            swap stream-copy        ] with-disposal    ] with-disposal ; ---=---Александр --
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421=/41014381___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Hooking the Back End

[Doug Coleman]

Hey Alexander,

We have a copy-file implemented with stream-copy already. It's non-blocking
and it might be fast enough, or we could optimize it. I haven't looked at
CopyFile for a long time, but maybe we shouldn't use that and should use
the default stream-copy version instead. Then you don't have to replace the
io-backend with a different one.

Doug


! From basis/io/directories/directories.factor
IN: io.directories

! Copying files
HOOK: copy-file io-backend ( from to -- ) ;

M: object copy-file
dup parent-directory make-directories
binary  [
swap binary  [
swap stream-copy
] with-disposal
] with-disposal ;


On Mon, Jun 13, 2016 at 3:31 PM, John Benediktsson  wrote:

> The delegate vocabulary might work. Also could initialed the Windows
> backend and then set yours and for all words you hart implemented just call
> the Windows one.
>
> > On Jun 13, 2016, at 11:28 AM, Alexander Ilin  wrote:
> >
> > Hello!
> >
> >  I'm having a bit of a fun problem, and I'd like you to help me with a
> solution.
> >
> >  I want to create a custom io-backend. In my implementation all calls to
> copy-file, move-file and delete-file should be captured and queued for
> deferred and/or background batch processing.
> >
> >  The problem: the current implementation of copy-file calls WinAPI's
> CopyFile, which blocks for the duration of the operation. For big files
> that takes time. I want to create my own implementation that would copy the
> file piece by piece in a Factor's cooperative non-blocking asynchronous
> background super-awesome thread.
> >
> >  The solution: I have created a custom backend singleton and intercepted
> the necessary methods. I'm advancing my implementation carefully one step
> at a time. Currently I'm making the copy-file calculate a sha-256 hash of
> the file, just to make sure it reads all of the contents correctly. Next
> step will be writing the data to the new location.
> >
> >  Already I have a problem: with-file-reader calls (file-reader)
> internally, which is hooked on io-backend. But my custom backend doesn't
> provide an implementation for (file-reader). The way I want this to work is
> this: my custom backend is to only intercept the methods I override
> (delete-file, copy-file and move-file), and all the other methods are to be
> handled by the previously installed backend. Kinda like an inheritance-type
> thing. Is there a way to do this?
> >
> >  Is there a better approach to the whole problem?
> >
> >  My code so far:
> >
> > USING: kernel namespaces
> >formatting prettyprint
> >checksums checksums.sha
> >io.backend io.files io.directories io.encodings.binary ;
> >
> > IN: batch
> >
> > SINGLETON: batch-io-backend
> >
> > : with-batch-io-backend ( quot -- )
> >[ batch-io-backend io-backend ] dip with-variable ; inline
> >
> > M: batch-io-backend delete-file ( path -- )
> >"TODO: batch delete file (%s)" sprintf . ;
> >
> > M: batch-io-backend copy-file ( from to -- )
> >2dup "batch copy file (%s -> %s)" sprintf .
> >drop binary [
> >sha-256 initialize-checksum-state
> >"hello" add-checksum-bytes
> >get-checksum .
> >] with-file-reader ;
> >
> > M: batch-io-backend move-file ( from to -- )
> >"TODO: batch move file (%s -> %s)" sprintf . ;
> >
> >
> > IN: scratchpad [ "README.md" "dst" copy-file ] with-batch-io-backend
> >
> > ---=---
> > Александр
> >
> >
> --
> > What NetFlow Analyzer can do for you? Monitors network bandwidth and
> traffic
> > patterns at an interface-level. Reveals which users, apps, and protocols
> are
> > consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> > J-Flow, sFlow and other flows. Make informed decisions using capacity
> > planning reports.
> https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> > ___
> > Factor-talk mailing list
> > Factor-talk@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/factor-talk
>
>
> --
> What NetFlow Analyzer can do for you? Monitors network bandwidth and
> traffic
> patterns at an interface-level. Reveals which users, apps, and protocols
> are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk
>
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and 

Re: [Factor-talk] Hooking the Back End

[John Benediktsson]

The delegate vocabulary might work. Also could initialed the Windows backend 
and then set yours and for all words you hart implemented just call the Windows 
one. 

> On Jun 13, 2016, at 11:28 AM, Alexander Ilin  wrote:
> 
> Hello!
> 
>  I'm having a bit of a fun problem, and I'd like you to help me with a 
> solution.
> 
>  I want to create a custom io-backend. In my implementation all calls to 
> copy-file, move-file and delete-file should be captured and queued for 
> deferred and/or background batch processing.
> 
>  The problem: the current implementation of copy-file calls WinAPI's 
> CopyFile, which blocks for the duration of the operation. For big files that 
> takes time. I want to create my own implementation that would copy the file 
> piece by piece in a Factor's cooperative non-blocking asynchronous background 
> super-awesome thread.
> 
>  The solution: I have created a custom backend singleton and intercepted the 
> necessary methods. I'm advancing my implementation carefully one step at a 
> time. Currently I'm making the copy-file calculate a sha-256 hash of the 
> file, just to make sure it reads all of the contents correctly. Next step 
> will be writing the data to the new location.
> 
>  Already I have a problem: with-file-reader calls (file-reader) internally, 
> which is hooked on io-backend. But my custom backend doesn't provide an 
> implementation for (file-reader). The way I want this to work is this: my 
> custom backend is to only intercept the methods I override (delete-file, 
> copy-file and move-file), and all the other methods are to be handled by the 
> previously installed backend. Kinda like an inheritance-type thing. Is there 
> a way to do this?
> 
>  Is there a better approach to the whole problem?
> 
>  My code so far:
> 
> USING: kernel namespaces
>formatting prettyprint
>checksums checksums.sha
>io.backend io.files io.directories io.encodings.binary ;
> 
> IN: batch
> 
> SINGLETON: batch-io-backend
> 
> : with-batch-io-backend ( quot -- )
>[ batch-io-backend io-backend ] dip with-variable ; inline
> 
> M: batch-io-backend delete-file ( path -- )
>"TODO: batch delete file (%s)" sprintf . ;
> 
> M: batch-io-backend copy-file ( from to -- )
>2dup "batch copy file (%s -> %s)" sprintf .
>drop binary [
>sha-256 initialize-checksum-state
>"hello" add-checksum-bytes
>get-checksum .
>] with-file-reader ;
> 
> M: batch-io-backend move-file ( from to -- )
>"TODO: batch move file (%s -> %s)" sprintf . ;
> 
> 
> IN: scratchpad [ "README.md" "dst" copy-file ] with-batch-io-backend
> 
> ---=---
> Александр
> 
> --
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity 
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk