Hi My streaming job uses a set of rules to process records from a stream. The rule set is defined in simple flat files, one rule per line. The rule set can change from time to time. A user will upload a new file that must replace the old rule set completely.
My problem is with reading and updating the rule set when I have a new one. I cannot update single rules. I need the whole rule set to validate it and build the internal representation to broadcast. I am reading the file with a *ContinuousFileReaderOperator* and *InputFormat* (via env.readFile(...) and creating the internal representation of the rule set I then broadcast. I get new files with processingMode = PROCESS_CONTINUOUSLY How do I know when I have read ALL the records from a physical file, to trigger validating and building the new Rule Set? I've been thinking about a processing-time trigger, waiting a reasonable time after I read the first rule of a new file, but it does not look safe if the user, for example, uploads two new files by mistake. Cheers Lorenzo
