Right, I would certainly agree about a single deployment of the package, but when it's pulling in and unpacked, I would expect we'd pull that locally, though of course that adds complexity, maybe some of storms local file caching could help us there. Still that I think is a discussion for another thread!
Simon > On 12 Apr 2017, at 18:48, Otto Fowler <ottobackwa...@gmail.com> wrote: > > What Casey and I had talked about ( and he actually implemented first with > stellar ) is having a single shared repository, without having to tie the > deployment > to the machine. > > This all still has to stand review so we’ll see. > Maybe Casey can comment. > > > >> On April 12, 2017 at 08:30:31, Simon Elliston Ball >> (si...@simonellistonball.com) wrote: >> >> Makes sense, however, would it make sense to unpack to local file system, >> which would then avoid further HDFS ops? >> >> >>> On 12 Apr 2017, at 13:24, Otto Fowler <ottobackwa...@gmail.com> wrote: >>> >>> "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are >>> deployed >>> if required. So the writes for HDFS are the unpacking the bundles ( with >>> the unpacked jars being loaded into a classloader ). >>> >>> If the unbundled is the same as the bundle, there is no op. In this case, >>> it is first run. >>> >>> So this is the parser bolt using the HDFS backed extension loader.” >>> >>> >>> This is part of the parser side loading, also feeds into the stellar load >>> from hdfs stuff. >>> >>> * metron has extensions >>> * extensions are packaged with their configurations and their ‘bundle’ ( >>> the nar part ) >>> * extension bundles are deployed to hdfs *lib directory >>> * parts of the system that use extensions discover and load them, with >>> class loaders setup for the bundle >>> * part of that, the bundles are unpackaged into a ‘working’ area if >>> required ( so if you drop a new version into the lib, the next app will >>> unpack the new version ) >>> * thus, the bundle system needs to be able to write to the working areas in >>> hdfs >>> * ??? >>> * profit >>> >>> >>>> On April 12, 2017 at 08:11:28, Simon Elliston Ball >>>> (si...@simonellistonball.com) wrote: >>>> >>>> I’m curious: Otto, what’s your use case for writing to HDFS from a >>>> parsers? >>>> >>>> Simon >>>> >>>> > On 12 Apr 2017, at 13:04, Justin Leet <justinjl...@gmail.com> wrote: >>>> > >>>> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group >>>> > and >>>> > with 775 will be able to write. >>>> > >>>> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dlyle65...@gmail.com> >>>> > wrote: >>>> > >>>> >> It's curious to me that you're writing directly from parsing, but I >>>> >> suspect >>>> >> that your parsing topology is running as the storm user and it can't >>>> >> write >>>> >> to those directories. >>>> >> >>>> >> -D... >>>> >> >>>> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ottobackwa...@gmail.com> >>>> >> wrote: >>>> >> >>>> >>> The indexing dir is created: >>>> >>> >>>> >>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir, >>>> >>> type="directory", >>>> >>> action="create_on_execute", >>>> >>> owner=self.__params.metron_user, >>>> >>> group=self.__params.metron_group, >>>> >>> mode=0775, >>>> >>> ) >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwa...@gmail.com) >>>> >>> wrote: >>>> >>> >>>> >>> >>>> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the >>>> >> following >>>> >>> exception: >>>> >>> >>>> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied: >>>> >>> user=storm, access=WRITE, >>>> >>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x >>>> >>> >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. >>>> >>> check(FSPermissionChecker.java:319) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. >>>> >>> check(FSPermissionChecker.java:292) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. >>>> >>> checkPermission(FSPermissionChecker.java:213) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. >>>> >>> checkPermission(FSPermissionChecker.java:190) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. >>>> >>> checkPermission(FSDirectory.java:1827) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. >>>> >>> checkPermission(FSDirectory.java:1811) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess( >>>> >>> >>>> >>> FSDirectory.java:1794) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs( >>>> >>> FSDirMkdirOp.java:71) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs( >>>> >>> FSNamesystem.java:4011) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. >>>> >>> mkdirs(NameNodeRpcServer.java:1102) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi >>>> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi >>>> >>> deTranslatorPB.java:630) >>>> >>> at >>>> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ >>>> >>> ClientNamenodeProtocol$2.callBlockingMethod( >>>> >> ClientNamenodeProtocolProtos. >>>> >>> java) >>>> >>> at >>>> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call( >>>> >>> >>>> >>> ProtobufRpcEngine.java:640) >>>> >>> >>>> >>> >>>> >>> The HDFS directory is created as such: >>>> >>> >>>> >>> self.__params.HdfsResource(self.__params.hdfs_metron_ >>>> >>> apps_extensions_working, >>>> >>> type="directory", >>>> >>> action="create_on_execute", >>>> >>> owner=self.__params.metron_user, >>>> >>> mode=0775) >>>> >>> >>>> >>> >>>> >>> As the hdfs write handlers I am logging in as such: >>>> >>> >>>> >>> HdfsSecurityUtil.login(stormConfig, fsConf); >>>> >>> FileSystem fileSystem = FileSystem.get(fsConf); >>>> >>> >>>> >>> I am not sure what is different from the indexing hdfs writer setup >>>> >>> here, >>>> >>> but what I’m doing obviously is not working. >>>> >>> >>>> >>> Any ideas? >>>> >>> >>>> >>> >>>> >>> - the branch: >>>> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy >>>> >>> >>>> >>> I am not up to date with master. >>>> >>> >>>> >> >>
