Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-03 Thread Ted Roche 
On Sun, Aug 2, 2020 at 7:25 PM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:

>
> Thanks for the ideas.  Honestly, it's the API that's the slow part of
> the whole mix.  That's the vendor's programmers; out of my control.
>

It's likely, if you can get in touch with a tech on the other end, that
they could send you a CSV dump file rather than bog down their API with a
lot of slow, single-record request.

OTOH, you might separate the acquisition task (the API) and storing the
result as an overnight process (so no one cares how long it takes) and do
the speedy stuff at a later point.


--- StripMime Report -- processed MIME parts ---
multipart/alternative
  text/plain (text body -- kept)
  text/html
---

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/CACW6n4ufS=qe5m0sjivyn8pu6nspdq8ndqdbm6upcacim32...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

RE: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-03 Thread Tracy Pearson 
Mike, 

There are SO many ways to do these processes.
VFP can launch other VFP COM objects and become multi-threaded.
VFP has a project that Bill mentioned earlier and I have never touched 
ParallelFox
C# has the ability to multi-thread and use a VFP COM object.

You're bottleneck is the API. Does the API also have a threshold that will 
block you for a time?
As an example of what I mean, the one of APIs I have written code to connect 
with only allow 5 calls a second. Making a 6th call in the same second will 
block the access token for 60 seconds. It's nice enough to inform me of the 
wait period I need when it is blocked. Testing proofed that. I keep that in 
mind when researching new APIs to use.

One thing I discovered with VFP and using the Winsock COM object to make calls 
to an API. The HTTP port will remain open for a short time. Even closing the 
COM object doesn't release the port. So a number of calls and the ports stay 
open for a period. We had a customer change a Registry setting to reduce the 
time the port will stay open and it helped with an odd accessing an API. I will 
need to dig up the support history to know what that registry key is, and the 
actual error.

Tracy


-Original Message-
From: ProfoxTech [mailto:profoxtech-boun...@leafe.com] On Behalf Of Ed Leafe
Sent: Monday, August 03, 2020 8:38 AM
To: profoxt...@leafe.com
Subject: Re: Task: Process thousands of records through a regular process -- 
multiple EXE runs to cut down processing time??

On Aug 2, 2020, at 18:24, MB Software Solutions, LLC 
 wrote:
> 
> Thanks for the ideas.  Honestly, it's the API that's the slow part of the 
> whole mix.  That's the vendor's programmers; out of my control.

How you call the API is under your control, though.

I don’t have the first idea how this would be done in VFP, but in general you 
get around a blocking task by using some async design. The most direct approach 
would be multiple threads, but some languages have a callback design. One of 
the reasons that the Go language became wildly popular over the past few years 
is that is async is a fundamental design in the language: 
https://medium.com/@gauravsingharoy/asynchronous-programming-with-go-546b96cd50c1.


-- Ed Leafe







[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/000c01d669a4$c8929150$59b7b3f0$@powerchurch.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-03 Thread Ed Leafe 
On Aug 2, 2020, at 18:24, MB Software Solutions, LLC 
 wrote:
> 
> Thanks for the ideas.  Honestly, it's the API that's the slow part of the 
> whole mix.  That's the vendor's programmers; out of my control.

How you call the API is under your control, though.

I don’t have the first idea how this would be done in VFP, but in general you 
get around a blocking task by using some async design. The most direct approach 
would be multiple threads, but some languages have a callback design. One of 
the reasons that the Go language became wildly popular over the past few years 
is that is async is a fundamental design in the language: 
https://medium.com/@gauravsingharoy/asynchronous-programming-with-go-546b96cd50c1.


-- Ed Leafe







___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/e3c0dc78-fa63-4f99-9300-4396ca75c...@leafe.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-03 Thread Charlie 
The bottom line is yes, your approach is reasonable. I have done pretty 
much the same approach myself.


To sum up: RLOCK() is fine, but I think I did an extra step like SKIP 
-1/SKIP (and consider the "refresh" setting). I don't recall the 
original thread, but I presume you're setting some kind of flag field to 
denote the record was processed already. Only set that after a 
successful RLOCK() of course.


I would suggest trying your set up on few hundred records, and put a bit 
of logging in there to describe what happens (simple strtofile() 
statements even). I think the benefit of number of processes running 
might max out based on the number of cores on the machine - e.g. a 
4-core machine may not benefit from running 16 .exe instances. But, on 
the other hand, if the API call is the thing taking the vast majority of 
time, it would seem like the local resources are mostly twiddling their 
thumbs. So running more than the core count may be beneficial.


But note that the API vendor may have their own throttle - maybe based 
on API key - to prevent someone from overloading their resources. And, 
of course, there are also horrible systems out there NoSQL datastores, 
bad Java, constrained NodeJS, just plain bad coding, blah blah, and it's 
hard to predict what kind of performance you'll get from them. So your 
logging may help find the API vendor's 'max performance' rate as well. 
When you hit that, like you imply, you will not be able to 'fix' it from 
your side.


-Charlie


On 8/2/2020 7:23 PM, MB Software Solutions, LLC wrote:
Right...we're using the API documentation from the vendor, doing it 
according to what they say.  It's not super quick but then again I 
don't think most people use it for processing tens of thousands of 
records quickly like my client's app does.  Most folks are using it to 
just get a single carrier's information (it's a trucking app) at a time.



On 8/2/2020 8:17 AM, António Tavares Lopes wrote:

Mike,

a) is it a Web API that you may call through an HTTP library?

b) and the parameters' values in each call to the API come from each 
record

in the table?

On Sat, Aug 1, 2020 at 6:07 PM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:

I've got a regular process that runs, basically using key 
information to

grab data from an API and then update the local VFP database. There are
maybe 64000 records to process, and each record to update through this
process takes about a second, so to process this group would take over
17 hours.  Each record could be processed on its own; there are no
relationships between each.

I don't want to start it and run the 64000 in a row for 17+ hours.  I
would like to design the app to use the table, RLOCK the row I'm
processing, and the UNLOCK the row when I'm done.  I figure with this
design, I could run multiple instances of the MyProgram.exe (similar to
how WestWind Web Connection allows you to run multiple instances) to
process the batch maybe 4x faster (if I launched 4 instances of
MyProgram.exe).  The basic construct would be as follows:

USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
process) i, tProcessed t, tError t, cSession c(10)
SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
  IF ProcessRecord(MyList.ID) THEN
  REPLACE tProcessed WITH DATETIME(), cSession WITH 
this.cSession

IN MyList
  ELSE
  REPLACE tError WITH DATETIME(), cSession WITH 
this.cSession IN

MyList
  ENDIF
ENDSCAN


Does anybody see any problems with that general design?  The
ProcessRecord method calls an API to get values and then updates the
local VFP record accordingly.

tia,
--Mike



--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus



[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/df9ed6dc-062e-aa17-8d6a-5963e7558...@gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-03 Thread António Tavares Lopes 
Depending on specifics, HTTP calls can be issued asynchronously. That would
mean, if applicable to your case, that the local process would not have to
wait for the API call to return in each iteration. Single-record processing
would be triggered upon completion of the respective API servicing, but
wouldn't stop the table scanning, at least completely. For instance, you
could set a number of concurrent calls - say 20, or some other number that
may be proven effective.

An example of this setup can be found in the overHere SDK (
https://github.com/atlopes/overHere). Calls to the Here location platform
"Geocoding Autocomplete API" are performed asynchronously. In the
AutoComplete-Async demo form, as the user keeps editing an address in the
textbox control, underlying calls to the platform web services are launched
and the retrieved suggestions displayed as fetched, while not preventing
the user to continue his/her editing.

The overHere SDK implements a callback mechanism that has to deal with
object persistence and XML over HTTP (instead of simple XMLHttp request),
but a simpler method could eventually be set in place for your scenario.

On Mon, Aug 3, 2020 at 12:23 AM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:

> Right...we're using the API documentation from the vendor, doing it
> according to what they say.  It's not super quick but then again I don't
> think most people use it for processing tens of thousands of records
> quickly like my client's app does.  Most folks are using it to just get
> a single carrier's information (it's a trucking app) at a time.
>
>
> On 8/2/2020 8:17 AM, António Tavares Lopes wrote:
> > Mike,
> >
> > a) is it a Web API that you may call through an HTTP library?
> >
> > b) and the parameters' values in each call to the API come from each
> record
> > in the table?
> >
> > On Sat, Aug 1, 2020 at 6:07 PM MB Software Solutions, LLC <
> > mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:
> >
> >> I've got a regular process that runs, basically using key information to
> >> grab data from an API and then update the local VFP database.  There are
> >> maybe 64000 records to process, and each record to update through this
> >> process takes about a second, so to process this group would take over
> >> 17 hours.  Each record could be processed on its own; there are no
> >> relationships between each.
> >>
> >> I don't want to start it and run the 64000 in a row for 17+ hours.  I
> >> would like to design the app to use the table, RLOCK the row I'm
> >> processing, and the UNLOCK the row when I'm done.  I figure with this
> >> design, I could run multiple instances of the MyProgram.exe (similar to
> >> how WestWind Web Connection allows you to run multiple instances) to
> >> process the batch maybe 4x faster (if I launched 4 instances of
> >> MyProgram.exe).  The basic construct would be as follows:
> >>
> >> USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
> >> process) i, tProcessed t, tError t, cSession c(10)
> >> SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
> >>   IF ProcessRecord(MyList.ID) THEN
> >>   REPLACE tProcessed WITH DATETIME(), cSession WITH
> this.cSession
> >> IN MyList
> >>   ELSE
> >>   REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
> >> MyList
> >>   ENDIF
> >> ENDSCAN
> >>
> >>
> >> Does anybody see any problems with that general design?  The
> >> ProcessRecord method calls an API to get values and then updates the
> >> local VFP record accordingly.
> >>
> >> tia,
> >> --Mike
> >>
> >>
> >>
> >> --
> >> This email has been checked for viruses by Avast antivirus software.
> >> https://www.avast.com/antivirus
> >>
> >>
[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/cadj74tgp5n8j2jkakiskeedw8cuetaskfnekhy8ovddfzov...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-02 Thread MB Software Solutions, LLC 

On 8/2/2020 11:45 AM, Ted Roche wrote:

This is a classic queue pattern: Make a task list of the items to be
processed (and perhaps breaking up multiple tasks for each item), and run
multiple workers against the list, updating the status as they go. As
Christof says, ultimately you have to LOCK, UPDATE, UNLOCK and think about
REFRESH.

64,000 records should take seconds, except for the slow parts. Focus on
those: is it a slow API? Is it setup and takedown that you could cache?

Think about the idea of separating the slow parts from the fast parts, so
your queue might have a slow part 1 and fast part 2 to be completed by
different executables. Then you could launch a dozen EXEs to process the
slow parts, improving the throughput, and a few EXEs to do the fast parts.



Thanks for the ideas.  Honestly, it's the API that's the slow part of 
the whole mix.  That's the vendor's programmers; out of my control.




--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/136d4927-c79e-cdc2-64d6-4fdd2ddba...@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-02 Thread MB Software Solutions, LLC 
Right...we're using the API documentation from the vendor, doing it 
according to what they say.  It's not super quick but then again I don't 
think most people use it for processing tens of thousands of records 
quickly like my client's app does.  Most folks are using it to just get 
a single carrier's information (it's a trucking app) at a time.



On 8/2/2020 8:17 AM, António Tavares Lopes wrote:

Mike,

a) is it a Web API that you may call through an HTTP library?

b) and the parameters' values in each call to the API come from each record
in the table?

On Sat, Aug 1, 2020 at 6:07 PM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:


I've got a regular process that runs, basically using key information to
grab data from an API and then update the local VFP database.  There are
maybe 64000 records to process, and each record to update through this
process takes about a second, so to process this group would take over
17 hours.  Each record could be processed on its own; there are no
relationships between each.

I don't want to start it and run the 64000 in a row for 17+ hours.  I
would like to design the app to use the table, RLOCK the row I'm
processing, and the UNLOCK the row when I'm done.  I figure with this
design, I could run multiple instances of the MyProgram.exe (similar to
how WestWind Web Connection allows you to run multiple instances) to
process the batch maybe 4x faster (if I launched 4 instances of
MyProgram.exe).  The basic construct would be as follows:

USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
process) i, tProcessed t, tError t, cSession c(10)
SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
  IF ProcessRecord(MyList.ID) THEN
  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
IN MyList
  ELSE
  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
MyList
  ENDIF
ENDSCAN


Does anybody see any problems with that general design?  The
ProcessRecord method calls an API to get values and then updates the
local VFP record accordingly.

tia,
--Mike



--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus



[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/7c001145-c35e-08ab-4542-06bc7a5a1...@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-02 Thread Bill Anderson 
Mike,

It sounds like you should look at ParallelFox...

Bill Anderson

On Sat, Aug 1, 2020 at 10:51 AM Eric Selje  wrote:

> I'm not sure I trust RLock(). I'd probably use a "Status" field that
> indicates the record is unprocessed, in process, or processed. This might
> be a good use case for Christof's DMult multi-threading utility
> .
>
>
> Eric
>
>
> On Sat, Aug 1, 2020 at 12:06 PM MB Software Solutions, LLC <
> mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:
>
> > I've got a regular process that runs, basically using key information to
> > grab data from an API and then update the local VFP database.  There are
> > maybe 64000 records to process, and each record to update through this
> > process takes about a second, so to process this group would take over
> > 17 hours.  Each record could be processed on its own; there are no
> > relationships between each.
> >
> > I don't want to start it and run the 64000 in a row for 17+ hours.  I
> > would like to design the app to use the table, RLOCK the row I'm
> > processing, and the UNLOCK the row when I'm done.  I figure with this
> > design, I could run multiple instances of the MyProgram.exe (similar to
> > how WestWind Web Connection allows you to run multiple instances) to
> > process the batch maybe 4x faster (if I launched 4 instances of
> > MyProgram.exe).  The basic construct would be as follows:
> >
> > USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
> > process) i, tProcessed t, tError t, cSession c(10)
> > SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
> >  IF ProcessRecord(MyList.ID) THEN
> >  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
> > IN MyList
> >  ELSE
> >  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
> > MyList
> >  ENDIF
> > ENDSCAN
> >
> >
> > Does anybody see any problems with that general design?  The
> > ProcessRecord method calls an API to get values and then updates the
> > local VFP record accordingly.
> >
> > tia,
> > --Mike
> >
> >
> >
> > --
> > This email has been checked for viruses by Avast antivirus software.
> > https://www.avast.com/antivirus
> >
> >
[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/ca+o1englyv0ritwb_z9wkyyd7pmv4naqdyck0cki_xcugju...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-02 Thread Ted Roche 
This is a classic queue pattern: Make a task list of the items to be
processed (and perhaps breaking up multiple tasks for each item), and run
multiple workers against the list, updating the status as they go. As
Christof says, ultimately you have to LOCK, UPDATE, UNLOCK and think about
REFRESH.

64,000 records should take seconds, except for the slow parts. Focus on
those: is it a slow API? Is it setup and takedown that you could cache?

Think about the idea of separating the slow parts from the fast parts, so
your queue might have a slow part 1 and fast part 2 to be completed by
different executables. Then you could launch a dozen EXEs to process the
slow parts, improving the throughput, and a few EXEs to do the fast parts.



On Sat, Aug 1, 2020 at 1:07 PM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:

> I've got a regular process that runs, basically using key information to
> grab data from an API and then update the local VFP database.  There are
> maybe 64000 records to process, and each record to update through this
> process takes about a second, so to process this group would take over
> 17 hours.  Each record could be processed on its own; there are no
> relationships between each.
>
> I don't want to start it and run the 64000 in a row for 17+ hours.  I
> would like to design the app to use the table, RLOCK the row I'm
> processing, and the UNLOCK the row when I'm done.  I figure with this
> design, I could run multiple instances of the MyProgram.exe (similar to
> how WestWind Web Connection allows you to run multiple instances) to
> process the batch maybe 4x faster (if I launched 4 instances of
> MyProgram.exe).  The basic construct would be as follows:
>
> USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
> process) i, tProcessed t, tError t, cSession c(10)
> SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
>  IF ProcessRecord(MyList.ID) THEN
>  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
> IN MyList
>  ELSE
>  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
> MyList
>  ENDIF
> ENDSCAN
>
>
> Does anybody see any problems with that general design?  The
> ProcessRecord method calls an API to get values and then updates the
> local VFP record accordingly.
>
> tia,
> --Mike
>
>
>
> --
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
>
[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/cacw6n4tns0vjmhw8idlkrld7p0mkv8e5h0t2wrwypgb3vh1...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-02 Thread António Tavares Lopes 
Mike,

a) is it a Web API that you may call through an HTTP library?

b) and the parameters' values in each call to the API come from each record
in the table?

On Sat, Aug 1, 2020 at 6:07 PM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:

> I've got a regular process that runs, basically using key information to
> grab data from an API and then update the local VFP database.  There are
> maybe 64000 records to process, and each record to update through this
> process takes about a second, so to process this group would take over
> 17 hours.  Each record could be processed on its own; there are no
> relationships between each.
>
> I don't want to start it and run the 64000 in a row for 17+ hours.  I
> would like to design the app to use the table, RLOCK the row I'm
> processing, and the UNLOCK the row when I'm done.  I figure with this
> design, I could run multiple instances of the MyProgram.exe (similar to
> how WestWind Web Connection allows you to run multiple instances) to
> process the batch maybe 4x faster (if I launched 4 instances of
> MyProgram.exe).  The basic construct would be as follows:
>
> USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
> process) i, tProcessed t, tError t, cSession c(10)
> SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
>  IF ProcessRecord(MyList.ID) THEN
>  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
> IN MyList
>  ELSE
>  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
> MyList
>  ENDIF
> ENDSCAN
>
>
> Does anybody see any problems with that general design?  The
> ProcessRecord method calls an API to get values and then updates the
> local VFP record accordingly.
>
> tia,
> --Mike
>
>
>
> --
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
>
[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/cadj74tg8so092gpfuf9ccd18n2fg+tu_jfosn13z_athoks...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-02 Thread MB Software Solutions, LLC 

Koen,

This is a question of theory.  I'm not going to lay out the entire set 
of procedures that get run here.  Presumptuous of you to make that 
"unacceptable" statement without knowing what actually happens behind 
the scenes.  I could be talking about a process that does 20 different 
things, checking this, that, and another thing, writing here, writing 
there.  As another mentioned, the bulk of time was the API (outside of 
my actual process), anyway.



On 8/1/2020 3:30 PM, Koen Piller wrote:

Also missing the code for your procedure ProcessRecord()

Op za 1 aug. 2020 om 20:05 schreef Koen Piller :


Hi,

64000 records in 17 hours That is way unacceptable. To run the code
you published would take normally around  minutes max. I suppose the most
of the time is not by your code but the API call to get the data into the
cursor ListOfRecsToProces, advise you to implement Codereference to find
the time consuming bottle-neck.
Instead of scan..endscan you may also try to make use of the SQL Update()
function, this should be faster.

Koen


Op za 1 aug. 2020 om 19:07 schreef MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com>:


I've got a regular process that runs, basically using key information to
grab data from an API and then update the local VFP database.  There are
maybe 64000 records to process, and each record to update through this
process takes about a second, so to process this group would take over
17 hours.  Each record could be processed on its own; there are no
relationships between each.

I don't want to start it and run the 64000 in a row for 17+ hours.  I
would like to design the app to use the table, RLOCK the row I'm
processing, and the UNLOCK the row when I'm done.  I figure with this
design, I could run multiple instances of the MyProgram.exe (similar to
how WestWind Web Connection allows you to run multiple instances) to
process the batch maybe 4x faster (if I launched 4 instances of
MyProgram.exe).  The basic construct would be as follows:

USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
process) i, tProcessed t, tError t, cSession c(10)
SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
  IF ProcessRecord(MyList.ID) THEN
  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
IN MyList
  ELSE
  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
MyList
  ENDIF
ENDSCAN


Does anybody see any problems with that general design?  The
ProcessRecord method calls an API to get values and then updates the
local VFP record accordingly.

tia,
--Mike



--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus



[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/20ca20df-6890-743e-5270-33dee0ec9...@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-01 Thread Stephen Russell 
What is your bottleneck in the process, an update of the backend data ?

Can your update or insert be made in groups of 100 or 1000 at a pass,
instead of one at a time?  Or do you need a key back for
additional processing?

On Sat, Aug 1, 2020 at 12:07 PM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:

> I've got a regular process that runs, basically using key information to
> grab data from an API and then update the local VFP database.  There are
> maybe 64000 records to process, and each record to update through this
> process takes about a second, so to process this group would take over
> 17 hours.  Each record could be processed on its own; there are no
> relationships between each.
>
> I don't want to start it and run the 64000 in a row for 17+ hours.  I
> would like to design the app to use the table, RLOCK the row I'm
> processing, and the UNLOCK the row when I'm done.  I figure with this
> design, I could run multiple instances of the MyProgram.exe (similar to
> how WestWind Web Connection allows you to run multiple instances) to
> process the batch maybe 4x faster (if I launched 4 instances of
> MyProgram.exe).  The basic construct would be as follows:
>
> USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
> process) i, tProcessed t, tError t, cSession c(10)
> SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
>  IF ProcessRecord(MyList.ID) THEN
>  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
> IN MyList
>  ELSE
>  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
> MyList
>  ENDIF
> ENDSCAN
>
>
> Does anybody see any problems with that general design?  The
> ProcessRecord method calls an API to get values and then updates the
> local VFP record accordingly.
>
> tia,
> --Mike
>
>
>
> --
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
>
[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/cajidmykzwsphe+euz6t5bgipvt-4a2abdbzvriyu13ooo2g...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-01 Thread Koen Piller 
Also missing the code for your procedure ProcessRecord()

Op za 1 aug. 2020 om 20:05 schreef Koen Piller :

> Hi,
>
> 64000 records in 17 hours That is way unacceptable. To run the code
> you published would take normally around  minutes max. I suppose the most
> of the time is not by your code but the API call to get the data into the
> cursor ListOfRecsToProces, advise you to implement Codereference to find
> the time consuming bottle-neck.
> Instead of scan..endscan you may also try to make use of the SQL Update()
> function, this should be faster.
>
> Koen
>
>
> Op za 1 aug. 2020 om 19:07 schreef MB Software Solutions, LLC <
> mbsoftwaresoluti...@mbsoftwaresolutions.com>:
>
>> I've got a regular process that runs, basically using key information to
>> grab data from an API and then update the local VFP database.  There are
>> maybe 64000 records to process, and each record to update through this
>> process takes about a second, so to process this group would take over
>> 17 hours.  Each record could be processed on its own; there are no
>> relationships between each.
>>
>> I don't want to start it and run the 64000 in a row for 17+ hours.  I
>> would like to design the app to use the table, RLOCK the row I'm
>> processing, and the UNLOCK the row when I'm done.  I figure with this
>> design, I could run multiple instances of the MyProgram.exe (similar to
>> how WestWind Web Connection allows you to run multiple instances) to
>> process the batch maybe 4x faster (if I launched 4 instances of
>> MyProgram.exe).  The basic construct would be as follows:
>>
>> USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
>> process) i, tProcessed t, tError t, cSession c(10)
>> SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
>>  IF ProcessRecord(MyList.ID) THEN
>>  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
>> IN MyList
>>  ELSE
>>  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
>> MyList
>>  ENDIF
>> ENDSCAN
>>
>>
>> Does anybody see any problems with that general design?  The
>> ProcessRecord method calls an API to get values and then updates the
>> local VFP record accordingly.
>>
>> tia,
>> --Mike
>>
>>
>>
>> --
>> This email has been checked for viruses by Avast antivirus software.
>> https://www.avast.com/antivirus
>>
>>
[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/cacuu1stgizctq6p8nedlvenqvd0ffnviuu5u9b0zdgsw+r3...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-01 Thread Koen Piller 
Hi,

64000 records in 17 hours That is way unacceptable. To run the code you
published would take normally around  minutes max. I suppose the most of
the time is not by your code but the API call to get the data into the
cursor ListOfRecsToProces, advise you to implement Codereference to find
the time consuming bottle-neck.
Instead of scan..endscan you may also try to make use of the SQL Update()
function, this should be faster.

Koen


Op za 1 aug. 2020 om 19:07 schreef MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com>:

> I've got a regular process that runs, basically using key information to
> grab data from an API and then update the local VFP database.  There are
> maybe 64000 records to process, and each record to update through this
> process takes about a second, so to process this group would take over
> 17 hours.  Each record could be processed on its own; there are no
> relationships between each.
>
> I don't want to start it and run the 64000 in a row for 17+ hours.  I
> would like to design the app to use the table, RLOCK the row I'm
> processing, and the UNLOCK the row when I'm done.  I figure with this
> design, I could run multiple instances of the MyProgram.exe (similar to
> how WestWind Web Connection allows you to run multiple instances) to
> process the batch maybe 4x faster (if I launched 4 instances of
> MyProgram.exe).  The basic construct would be as follows:
>
> USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
> process) i, tProcessed t, tError t, cSession c(10)
> SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
>  IF ProcessRecord(MyList.ID) THEN
>  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
> IN MyList
>  ELSE
>  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
> MyList
>  ENDIF
> ENDSCAN
>
>
> Does anybody see any problems with that general design?  The
> ProcessRecord method calls an API to get values and then updates the
> local VFP record accordingly.
>
> tia,
> --Mike
>
>
>
> --
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
>
[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/cacuu1ssh22opok_5iv2jfvbcacxyo2lansj_t0fryobc5ud...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-01 Thread Christof Wollenhaupt 
> I'm not sure I trust RLock(). I'd probably use a "Status" field that
> indicates the record is unprocessed, in process, or processed. This might
> be a good use case for Christof's DMult multi-threading utility

You still have to use either RLOCK() or a SET REFRESH value of -1 to read the 
status field, because otherwise VFP will lie to you about the current value of 
that field. Unless you move away from a record, you get the same value for 
hours in VFP. Even if you move away, the cache is considered to be the truth 
for up to how many seconds SET REFRESH indicates.

Christof


___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/8bc9aee1-ed4a-43b2-b3db-6bf3c5b91...@wollenhaupt.org
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Re: Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-01 Thread Eric Selje 
I'm not sure I trust RLock(). I'd probably use a "Status" field that
indicates the record is unprocessed, in process, or processed. This might
be a good use case for Christof's DMult multi-threading utility
.


Eric


On Sat, Aug 1, 2020 at 12:06 PM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:

> I've got a regular process that runs, basically using key information to
> grab data from an API and then update the local VFP database.  There are
> maybe 64000 records to process, and each record to update through this
> process takes about a second, so to process this group would take over
> 17 hours.  Each record could be processed on its own; there are no
> relationships between each.
>
> I don't want to start it and run the 64000 in a row for 17+ hours.  I
> would like to design the app to use the table, RLOCK the row I'm
> processing, and the UNLOCK the row when I'm done.  I figure with this
> design, I could run multiple instances of the MyProgram.exe (similar to
> how WestWind Web Connection allows you to run multiple instances) to
> process the batch maybe 4x faster (if I launched 4 instances of
> MyProgram.exe).  The basic construct would be as follows:
>
> USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
> process) i, tProcessed t, tError t, cSession c(10)
> SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
>  IF ProcessRecord(MyList.ID) THEN
>  REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
> IN MyList
>  ELSE
>  REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
> MyList
>  ENDIF
> ENDSCAN
>
>
> Does anybody see any problems with that general design?  The
> ProcessRecord method calls an API to get values and then updates the
> local VFP record accordingly.
>
> tia,
> --Mike
>
>
>
> --
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
>
[excessive quoting removed by server]

___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/CAAwxvUk7hukD6y=7caccuyydaccjoyqy15b5_h1nwajarbu...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.

Task: Process thousands of records through a regular process -- multiple EXE runs to cut down processing time??

2020-08-01 Thread MB Software Solutions, LLC 
I've got a regular process that runs, basically using key information to 
grab data from an API and then update the local VFP database.  There are 
maybe 64000 records to process, and each record to update through this 
process takes about a second, so to process this group would take over 
17 hours.  Each record could be processed on its own; there are no 
relationships between each.


I don't want to start it and run the 64000 in a row for 17+ hours.  I 
would like to design the app to use the table, RLOCK the row I'm 
processing, and the UNLOCK the row when I'm done.  I figure with this 
design, I could run multiple instances of the MyProgram.exe (similar to 
how WestWind Web Connection allows you to run multiple instances) to 
process the batch maybe 4x faster (if I launched 4 instances of 
MyProgram.exe).  The basic construct would be as follows:


USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to 
process) i, tProcessed t, tError t, cSession c(10)

SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
    IF ProcessRecord(MyList.ID) THEN
        REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession 
IN MyList

    ELSE
        REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN 
MyList

    ENDIF
ENDSCAN


Does anybody see any problems with that general design?  The 
ProcessRecord method calls an API to get values and then updates the 
local VFP record accordingly.


tia,
--Mike



--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


___
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message: 
https://leafe.com/archives/byMID/1b438d3a-0117-f793-4792-1d63a4245...@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the 
author, and do not constitute legal or medical advice. This statement is added 
to the messages for those lawyers who are too stupid to see the obvious.