Andrew,
On Wed, Feb 22, 2017 at 11:21 AM, Andrew Grande <[email protected]> wrote: > I am observing one assumption in this thread. For some reason we are > implying all these will be hadoop compatible file systems. They don't > always have an HDFS plugin, nor should they as a mandatory requirement. > You are partially correct. There is a direct assumption in the availability of a HCFS (thanks Matt!) implementation. This is the case with: * Windows Azure Blob Storage * Google Cloud Storage Connector * MapR FileSystem (currently done via NAR recompilation / mvn profile) * Alluxio * Isilon (via HDFS) * others But I would't say this will apply to every other use storage system and in certain cases may not even be necessary (e.g. Isilon scale-out storage may be reached using its native HDFS compatible interfaces). Untie completely from the Hadoop nar. This allows for effective minifi > interaction without the weight of hadoop libs for example. Massive size > savings where it matters. > > Are you suggesting a use case were MiNiFi agents interact directly with cloud storage, without relying on NiFi hubs to do that? > For the deployment, it's easy enough for an admin to either rely on a > standard tar or rpm if the NAR modules are already available in the distro > (well, I won't talk registry till it arrives). Mounting a common directory > on every node or distributing additional jars everywhere, plus configs, and > then keeping it consistent across is something which can be avoided by > simpler packaging. > As long the NAR or RPM supports your use-case, which is not the case of people running NiFi with MapR-FS for example. For those, a recompilation is required anyway. A flexible processor may remove the need to recompile (I am currently playing with the classpath implication to MapR users). Cheers
