Hi, See my comments below
Thanks /Lennart > -----Original Message----- > From: Hans Feldt > Sent: den 5 juli 2013 17:12 > To: Lennart Lund; Anders Widell > Cc: [email protected]; [email protected] > Subject: RE: [devel] [PATCH 0 of 7] Review request: logsv: Fix hanging main > thread when file i/o don't return > > > > -----Original Message----- > > From: Lennart Lund > > Sent: den 5 juli 2013 16:51 > > To: Hans Feldt; Anders Widell > > Cc: [email protected]; [email protected]; > > Lennart Lund > > Subject: RE: [devel] [PATCH 0 of 7] Review request: logsv: Fix hanging > > main thread when file i/o don't return > > > > In principle I agree with Hans that the best way of implementing inter > > thread communication is to use a common design pattern (in OpenSAF > > LEAP mailbox). This is for example if there are two threads running in > parallel that needs to communicate and the main synchronization is one > thread waiting for a message from the other thread, but this is not the case > in here. > > > > In order to protect the log server "main thread" (MT) from hanging if > > a file operation like write, mkdir etc. does not return, all such operations > are done in a separate "file thread" (FT). > > Yes fine > > > Functions running in the "Main Thread" (MT) that needs file system > > operations handle over the execution to the FT when file handling has to > be done. > > Execution is then given back to the MT again. If a file operation does > > not return FT will hang but MT will time out the FT and resume. A timeout > will be handled as a file operation fail. > > The MT can detect if the FT is hanging and new requests for file operations > will be "failed". > > The mailbox between the mds thread and the main thread is limited and size > and intended for server protection. The important thing here is that > messages entered the mailbox should not be discarded. Only new ones once > the mailbox max limit is reached. With the extra fs thread the same > semantics needs to be provided. The main thread cannot just discard > messages that earlier was allowed to enter the mailbox. > > Please explain how this will be handled. The current log implementation: --------------------------------------- Overload protection in the Log Server (LGS) is a message queue used between the mds thread and the LGS thread where messages from the LGA are stored until the log server has time to handle them (meaning that messages e.g. log records can be sent to LGS faster than LGS can handle them without loss). Thresholds can be set that protects the queue from using up all available memory. If a threshold limit is reached the client gets information about the situation. In "normal" operation (everything is started, streams are created etc.) the log service normally waits for log records. If a log record is sent from LGA the server (LGS) wakes up on the event that there is a message in the queue. The message (log record) will be popped of the queue. The server will handle the record and write it to the corresponding file on the file system. If everything went well a callback for the client will be invoked in order to give the client a SA_AIS_OK message for this write request (an asynchronous operation). But... If writing the record fails e.g. the write operation returns an error the log record will be lost (it was removed from the message queue before the write). LGS will invoke the callback with a SA_AIS_ERR_TRY_AGAIN message. If the write operation because of some problem with the file system does not return the log server will "hang" and no more messages will be served. If LGS "hangs" long enough the node will be restarted because of the LGS not handling health check. Before that happen the message queue may eventually reach the (queue full) threshold. A client waiting for a callback may timeout. If the write operation eventually does return the server will invoke the callback with a message depending on the reply from the write operation as described above. The log record may have been lost. If the file system is not working any file operation may "get stuck" meaning that the LGS will "hang". This means for example that LGS may "hang" already during initialization (when creating/opening alarm, notification and system streams) causing the node to be restarted. With the #9 enhancement: --------------------------------- The overload protection mechanism is not changed at all. It still makes it possible to send log records faster than the LGS can handle them and protects the message queue from using too much memory in the same way as before. This enhancement is meant to protect the LGS from hanging if any (POSIX) file operation does not return. This in order to remove the problem with node restarts and no answer to clients invoking operations towards the log service. This means some new behavior: - No synchronous operations will be blocked because of a "hanging" file operation including opening of streams. Opening of a stream means that file operations has to be done e.g. files created, in some cases creation of new directories, etc. If this is not possible during stream creation it will be done when trying to write records to the stream. - As before a request to write a log record has to be fetched from the message queue when it shall be handled by the LGS. If writing the log record fails the situation will be handled as before i.e. the callback will be invoked with a SA_AIS_ERR_TRY_AGAIN message. The only difference is that if the (POSIX) operation (write) does not return the log service will not "hang" indefinitely instead it will resume execution after a timeout. Such a timeout will be handled as if the (POSIX) file system handling has failed and therefore result in a callback with a SA_AIS_ERR_TRY_AGAIN message. In this situation we do not know if the file operation will eventually succeed and the log record is written or if it is lost. If a new log record is sent e.g. because of the Try Again message it will be fetched from the queue and if the FT is still hanging no attempt to write the record is done. The callback will be invoked with Try Again message and the log record is discarded. If a log record is sent w hen the file system is working again it will be written and SA_AIS_OK is reported in the callback. This means that in worst case a log record can be saved twice (if the first try succeeded after the timeout occurred). Weakness: - A log record may be written twice (this is probably a smaller problem than losing a record), see above. - There is no recovery mechanism if the FT still "hangs" after the file system has start to work again (the FT thread should maybe be restarted after some time in such a situation?). - The client will get Try Again messages as long as the file system is hanging. Maybe a SA_AIS_ERR_TIMEOUT message shall be sent instead if the FT is timed out (or already hanging). > > > > > The main "communication" between the MT and the FT is synchronization. > > The MT is suspended, FT takes over execution (timed out if hanging), > > MT is resumed etc. This is handled with condition variable, mutex and > > synchronization flags. There is also some parameters that has to be > > sent from MT to FT and from FT to MT (this is done very simple). This > > parameter handling must also be part of the synchronization, e.g there > > must be no risk of that a mutex is locked if the FT hangs. LEAP > > mailbox is not suitable for this handling. The > > A mailbox limited in size would achieve the same result I guess I order to be sure I have to investigate this in more detail. However the synchronization between the threads is not message based. I probably still need to use condition variable and therefore also mutex. However the current solution is working. > > > code for doing this is not very complicated. Everything is done within two > functions (file_hndl_thread and log_file_api) in file lgs_file.c. > > The solution is discussed an confirmed by Anders Widell. > > Well he is not the opensaf community and I used to be LOG maintainer so ... What I mean is that I have discussed and confirmed that the solution for thread synchronization I have used is technically sound and follows "normal" ways (in general) of handling similar situations. > > /Hans > > > > > /Lennart > > > > > > > -----Original Message----- > > > From: Hans Feldt > > > Sent: den 5 juli 2013 11:59 > > > To: Lennart Lund > > > Cc: [email protected]; > > > [email protected] > > > Subject: RE: [devel] [PATCH 0 of 7] Review request: logsv: Fix > > > hanging main thread when file i/o don't return > > > > > > The LEAP mailbox is not "overkill", it removes the need for locks > > > etc in each thread. It is a design pattern used by all services > > > including LOG itself. Breaking such pattern I don't think is OK for > > > zero benefit. Likely the other way around, introducing new problems and > more code for locking etc. > > > /Hans > > > > > > > -----Original Message----- > > > > From: Lennart Lund > > > > Sent: den 5 juli 2013 09:47 > > > > To: Hans Feldt > > > > Cc: [email protected]; > > > > [email protected] > > > > Subject: RE: [devel] [PATCH 0 of 7] Review request: logsv: Fix > > > > hanging main thread when file i/o don't return > > > > > > > > In this case is the messages are sent in both directions. However > > > > it is very "simple" messages and no message queue is needed. To > > > > use LEAP > > > mailbox would be "overkill" > > > > > > > > /Lennart > > > > > > > > > -----Original Message----- > > > > > From: Hans Feldt > > > > > Sent: den 20 juni 2013 13:46 > > > > > To: Lennart Lund > > > > > Cc: [email protected]; > > > > > [email protected] > > > > > Subject: Re: [devel] [PATCH 0 of 7] Review request: logsv: Fix > > > > > hanging main thread when file i/o don't return > > > > > > > > > > Why isn't the same design pattern used as in all other services > > > > > in opensaf - a LEAP mailbox for *message passing* between threads? > > > > > > > > > > Thanks, > > > > > Hans > > > > > > > > > > On 06/18/2013 02:54 PM, Lennart Lund wrote: > > > > > > Summary: logsv: Fix hanging main thread when file i/o don't > > > > > > return Review request for Trac Ticket(s): #9 Peer Reviewer(s): > > > > > > Madhurika Koppula, (Anders Widell, Hans Feldt) Pull request > > > > > > to: NA Affected > > > > > > branch(es): devel (4.4) Development branch: <<IF ANY GIVE THE > > > > > > REPO > > > > > > URL>> > > > > > > > > > > > > -------------------------------- > > > > > > Impacted area Impact y/n > > > > > > -------------------------------- > > > > > > Docs n > > > > > > Build system n > > > > > > RPM/packaging n > > > > > > Configuration files n > > > > > > Startup scripts n > > > > > > SAF services y > > > > > > OpenSAF services n > > > > > > Core libraries n > > > > > > Samples n > > > > > > Tests n > > > > > > Other n > > > > > > > > > > > > > > > > > > Comments (indicate scope for each "y" above): > > > > > > --------------------------------------------- > > > > > > All file operations are handled in a separate thread. > > > > > > Functions doing operations on file system are changed so that > > > > > > this operations are done by a "handler" that is running in a > > > > > > separate thread. If a file operation "hangs" it will be timed > > > > > > out and an error is reported back to the > > > > > main thread. > > > > > > > > > > > > NOTE: > > > > > > In order to simplify retesting and troubleshooting some test > > > > > > traces and test code (tagged with LLDTEST) is not yet removed. > > > > > > This will be > > > > > removed or changed to "correct" > > > > > > TRACEs and LOGs before pushing. > > > > > > > > > > > > changeset 2296ad137a4f783c3efc69adf8db7261697aa327 > > > > > > Author: Lennart Lund <[email protected]> > > > > > > Date: Tue, 18 Jun 2013 12:35:03 +0200 > > > > > > > > > > > > logsv: Fix hanging main thread when file i/o don't return. > > > > > > Part 1 > > > > > > > > > > > > Generic thread handling: > > > > > > - Generic thread handling > > > > > > - Test handlers and two "real" handlers implemented > > > > > > > > > > > > changeset a6c505373ce8925fe2b8bf100c864b27d7f14b72 > > > > > > Author: Lennart Lund <[email protected]> > > > > > > Date: Tue, 18 Jun 2013 12:35:52 +0200 > > > > > > > > > > > > logsv: Fix hanging main thread when file i/o don't return. > > > > > > Part 2 > > > > > > > > > > > > More functions converted to use threaded file handling. > > > > > > - get_number_of_log_files_hdl() > > > > > > - Cleaning up init handling > > > > > > > > > > > > changeset e7119938da9f307f20aab9894934be08a122941f > > > > > > Author: Lennart Lund <[email protected]> > > > > > > Date: Tue, 18 Jun 2013 12:39:00 +0200 > > > > > > > > > > > > logsv: Fix hanging main thread when file i/o don't return. > > > > > > Part 3 > > > > > > > > > > > > More functions converted to use threaded file handling. > > > > > > - write log record function > > > > > > > > > > > > changeset 0574270ab6bf3a65548a8d23ec2989e9f26d67aa > > > > > > Author: Lennart Lund <[email protected]> > > > > > > Date: Tue, 18 Jun 2013 12:45:34 +0200 > > > > > > > > > > > > logsv: Fix hanging main thread when file i/o don't return. > > > > > > Part 4 > > > > > > > > > > > > More functions converted to use threaded file handling. > > > > > > - create_config_file_hdl > > > > > > > > > > > > changeset b989e158e38afa8fc231f40d0a32262611a334d2 > > > > > > Author: Lennart Lund <[email protected]> > > > > > > Date: Tue, 18 Jun 2013 12:45:49 +0200 > > > > > > > > > > > > logsv: Fix hanging main thread when file i/o don't return. > > > > > > Part 5 > > > > > > > > > > > > More functions converted to use threaded file handling. > > > > > > - lgs_file_rename_hfop(..) > > > > > > > > > > > > changeset aa83514345401e13c648d7d78940fc3b50f05636 > > > > > > Author: Lennart Lund <[email protected]> > > > > > > Date: Tue, 18 Jun 2013 12:45:50 +0200 > > > > > > > > > > > > logsv: Fix hanging main thread when file i/o don't return. > > > > > > Part 6 > > > > > > > > > > > > More functions converted to use threaded file handling: > > > > > > - check_path_exists_hdl(..) > > > > > > - This is the first inc for error handling updates. > > > > > > > > > > > > changeset 54b294afdfdc79c3fda2ef925939fc82633ce6ab > > > > > > Author: Lennart Lund <[email protected]> > > > > > > Date: Tue, 18 Jun 2013 12:45:50 +0200 > > > > > > > > > > > > logsv: Fix hanging main thread when file i/o don't return. > > > > > > Part 7 > > > > > > > > > > > > - Handling of object implementer rejects > > > > > > - Invalidate stream fd if errno EBADF when writing log record > > > > > > - Fix Error handling for too long path (> PATH_MAX) > > > > > > - Rename function that got temporary names during devel > > > > > > - Functions that uses a handler in file thread has got > > > > > > extension _h > > > > > > > > > > > > > > > > > > Added Files: > > > > > > ------------ > > > > > > README_LOGENH > > > > > > osaf/services/saf/logsv/lgs/lgs_file.c > > > > > > osaf/services/saf/logsv/lgs/lgs_file.h > > > > > > osaf/services/saf/logsv/lgs/lgs_filehdl.c > > > > > > osaf/services/saf/logsv/lgs/lgs_filehdl.h > > > > > > > > > > > > > > > > > > Complete diffstat: > > > > > > ------------------ > > > > > > osaf/services/saf/logsv/lgs/Makefile.am | 8 +- > > > > > > osaf/services/saf/logsv/lgs/lgs.h | 3 +- > > > > > > osaf/services/saf/logsv/lgs/lgs_amf.c | 10 +- > > > > > > osaf/services/saf/logsv/lgs/lgs_cb.h | 2 + > > > > > > osaf/services/saf/logsv/lgs/lgs_evt.c | 30 +++--- > > > > > > osaf/services/saf/logsv/lgs/lgs_evt.h | 4 + > > > > > > osaf/services/saf/logsv/lgs/lgs_file.c | 421 > > > > > > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > +++++++++++++++++++++++++++++++++++ > > > > > > osaf/services/saf/logsv/lgs/lgs_file.h | 121 > > > > > ++++++++++++++++++++++++++ > > > > > > osaf/services/saf/logsv/lgs/lgs_filehdl.c | 377 > > > > > > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > +++++++++++++++++++++++++ > > > > > > osaf/services/saf/logsv/lgs/lgs_filehdl.h | 30 ++++++ > > > > > > osaf/services/saf/logsv/lgs/lgs_imm.c | 87 > > > > > > +++++++++++------- > > > > > > osaf/services/saf/logsv/lgs/lgs_main.c | 14 ++- > > > > > > osaf/services/saf/logsv/lgs/lgs_mbcsv.c | 16 +- > > > > > > osaf/services/saf/logsv/lgs/lgs_stream.c | 338 > > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++----- > ---- > > > ----- > > > > > ---------- > > > > > > osaf/services/saf/logsv/lgs/lgs_stream.h | 27 +++-- > > > > > > osaf/services/saf/logsv/lgs/lgs_util.c | 105 > > > ++++++++++++++++++++-- > > > > > > osaf/services/saf/logsv/lgs/lgs_util.h | 15 ++- > > > > > > 17 files changed, 1401 insertions(+), 207 deletions(-) > > > > > > > > > > > > > > > > > > Testing Commands: > > > > > > ----------------- > > > > > > logtest can be used to run a regression test. > > > > > > > > > > > > > > > > > > Testing, Expected Results: > > > > > > -------------------------- > > > > > > All tests shall pass. > > > > > > If the file handler thread is "hanging" then > > > > > > - tests that writes log records will fail > > > > > > - IMM configuration of log related objects that requires > > > > > > access to file system will fail > > > > > > > > > > > > > > > > > > Conditions of Submission: > > > > > > ------------------------- > > > > > > Ack by Madhurika Koppula > > > > > > > > > > > > > > > > > > Arch Built Started Linux distro > > > > > > ------------------------------------------- > > > > > > mips n n > > > > > > mips64 n n > > > > > > x86 n n > > > > > > x86_64 n n > > > > > > powerpc n n > > > > > > powerpc64 n n > > > > > > > > > > > > > > > > > > Reviewer Checklist: > > > > > > ------------------- > > > > > > [Submitters: make sure that your review doesn't trigger any > > > > > > checkmarks!] > > > > > > > > > > > > > > > > > > Your checkin has not passed review because (see checked entries): > > > > > > > > > > > > ___ Your RR template is generally incomplete; it has too many > > > > > > blank > > > entries > > > > > > that need proper data filled in. > > > > > > > > > > > > ___ You have failed to nominate the proper persons for review > > > > > > and > > > push. > > > > > > > > > > > > ___ Your patches do not have proper short+long header > > > > > > > > > > > > ___ You have grammar/spelling in your header that is unacceptable. > > > > > > > > > > > > ___ You have exceeded a sensible line length in your > > > > > headers/comments/text. > > > > > > > > > > > > ___ You have failed to put in a proper Trac Ticket # into your > commits. > > > > > > > > > > > > ___ You have incorrectly put/left internal data in your > comments/files > > > > > > (i.e. internal bug tracking tool IDs, product names etc) > > > > > > > > > > > > ___ You have not given any evidence of testing beyond basic > > > > > > build > > > tests. > > > > > > Demonstrate some level of runtime or other sanity testing. > > > > > > > > > > > > ___ You have ^M present in some of your files. These have to > > > > > > be > > > removed. > > > > > > > > > > > > ___ You have needlessly changed whitespace or added whitespace > > > crimes > > > > > > like trailing spaces, or spaces before tabs. > > > > > > > > > > > > ___ You have mixed real technical changes with whitespace and > other > > > > > > cosmetic code cleanup changes. These have to be separate > commits. > > > > > > > > > > > > ___ You need to refactor your submission into logical chunks; there > is > > > > > > too much content into a single commit. > > > > > > > > > > > > ___ You have extraneous garbage in your review (merge commits > > > > > > etc) > > > > > > > > > > > > ___ You have giant attachments which should never have been > sent; > > > > > > Instead you should place your content in a public tree to be > pulled. > > > > > > > > > > > > ___ You have too many commits attached to an e-mail; resend as > > > threaded > > > > > > commits, or place in a public tree for a pull. > > > > > > > > > > > > ___ You have resent this content multiple times without a > > > > > > clear > > > indication > > > > > > of what has changed between each re-send. > > > > > > > > > > > > ___ You have failed to adequately and individually address all of > > > > > > the > > > > > > comments and change requests that were proposed in the > > > > > > initial > > > review. > > > > > > > > > > > > ___ You have a misconfigured ~/.hgrc file (i.e. username, > > > > > > email > > > > > > etc) > > > > > > > > > > > > ___ Your computer have a badly configured date and time; > > > > > > confusing > > > the > > > > > > the threaded patch review. > > > > > > > > > > > > ___ Your changes affect IPC mechanism, and you don't present > > > > > > any > > > results > > > > > > for in-service upgradability test. > > > > > > > > > > > > ___ Your changes affect user manual and documentation, your > > > > > > patch > > > series > > > > > > do not contain the patch that updates the Doxygen manual. > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > ---- > > > > > > ---- > > > > > > -------- This SF.net email is sponsored by Windows: > > > > > > > > > > > > Build for Windows Store. > > > > > > > > > > > > http://p.sf.net/sfu/windows-dev2dev > > > > > > _______________________________________________ > > > > > > Opensaf-devel mailing list > > > > > > [email protected] > > > > > > https://lists.sourceforge.net/lists/listinfo/opensaf-devel > > > > > > > > > > > > ------------------------------------------------------------------------------ This SF.net email is sponsored by Windows: Build for Windows Store. http://p.sf.net/sfu/windows-dev2dev _______________________________________________ Opensaf-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-devel
