Hi, My use case is that I want to ship a load of rows from an RDMS periodically and put in HDFS as Avro. QueryTable processor has functionality that would be great i.e. maxcolumn value (there are couple of columns I could use for this from the data) and it is this functionality I am looking for, BUT the data is not from one single table. The nature of the RDBMS is that the business view on the data requires a bunch of joins from other tables/schemas to get the correct Avro file so the options I appear to have are
a) Use QueryTable for each table that make up the business view and do the joins etc. in HDFS (Spark or something) – or potentially do the reconciliation within NiFi??? b) Use ExecuteSQL to run the complete SQL to get the rows which can easily be put into HDFS as Avro given that the line will be the business (denormalised) data that is required. The problem with a) is the reconciliation (denormalisation) of the data and the problem with b) is how to maintain the maxcolumn value so I only get the data since the last run. In order to address b) can I use the DistrubutedMapCacheServer & Client to hold a key/value pair of last run date and extract from this date as a parameter? Thanks for any suggestions. Conrad SecureData, combating cyber threats ______________________________________________________________________ The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote. SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
