Hi,
Emitting file information for a file based source like a file input operator in malhar seems like a good feature to provide. It is useful information for any downstream operator to know that a data tuple belongs to a certain file for instance . We propose to add capability in the abstract file input operator to emit file control tuples. These control tuples can include filenames as well as any metadata that the user wishes to include along with it. To link this meta data to each tuple, we can add another port to the input operator which would carry the meta data along with the actual tuple. We can try to reduce the amount of meta data that goes with each tuple by having some sort of meta encoding in the control tuple. ~ Bhupesh _______________________________________________________ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org
