Thanks Yakov. I think this is more subtle. Our loading via IgniteDatastreamer is idempotent, but this depends on being certain that a batch of work has successfully completed. It is *not* sufficient for us to listen to the futures returned by addData, then to call flush(), and then to record success if there have been no exceptions. We must wait until get() is called on every future before recording that the operation was successful. The fact that the future is done is not sufficient, we need to know that it is done and there is no exception. We can call flush and then do a future.get() on the incomplete futures, but not as an assert. (It is valid to assert that fut.isDone(), but that is not sufficient.)
Based on by current understanding, I think this is a flaw in Ignite, even if the fix might only be to clarify the comments for flush() to make this behavior clear. -DH On Wed, Apr 25, 2018 at 5:43 PM, Yakov Zhdanov <[email protected]> wrote: > David, can you try adding all the futures to a collection and change > assert condition to check that for all futures in that collection > fut.isDone() is true? > > This should be a proper assertion. Please let me know if it works. > > Yakov Zhdanov > > чт, 26 апр. 2018 г., 0:30 David Harvey <[email protected]>: > >> We had assumed that if we did, in one thread: >> IgniteFuture<?> f = streamer.addData(...); >> f.listen(...) >> >> streamer.flush(); >> >> assert( all the prior futures have completed); << this triggered. >> >> >> >> I can't determine if this a bug, or just that the description could use >> improvement. >> >> Inspecting the code it looks like the thread issuing the flush can be >> awoken prior to the apply function for the addData future is called, since >> GridFutureAdapter.unblockAll() will unpark or notify in a arbitrary >> order. >> >> >> *Disclaimer* >> >> The information contained in this communication from the sender is >> confidential. It is intended solely for use by the recipient and others >> authorized to receive it. If you are not the recipient, you are hereby >> notified that any disclosure, copying, distribution or taking action in >> relation of the contents of this information is strictly prohibited and may >> be unlawful. >> >> This email has been scanned for viruses and malware, and may have been >> automatically archived by *Mimecast Ltd*, an innovator in Software as a >> Service (SaaS) for business. Providing a *safer* and *more useful* place >> for your human generated data. Specializing in; Security, archiving and >> compliance. To find out more Click Here >> <http://www.mimecast.com/products/>. >> > Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
