Thanks Justin, I’m working on rebasing right now, then I’ll be able to try it out.
On April 12, 2017 at 08:26:08, Justin Leet (justinjl...@gmail.com) wrote: We changed the HDFS owner of /apps/metron/indexing/indexed a bit ago to be metron:hadoop, specifically because Storm wasn't writing to HDFS (and had perms issues). If you look into the indexing output directories, my expectation is that you aren't getting any data out (and the Storm indexing topology is throwing permissions errors). PR for that change: https://github.com/apache/incubator-metron/pull/488 Your branch has the old code: https://github.com/ottobackwards/incubator-metron/blob/parser_deploy/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py#L98 New code: https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py To set up ownership on startup with the hadoop group after you pull in master (or at least make the changes from that PR) self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir, type="directory", action="create_on_execute", owner=self.__params.metron_user, group=self.__params.hadoop_group, mode=0775, ) Justin On Wed, Apr 12, 2017 at 8:24 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are > deployed > if required. So the writes for HDFS are the unpacking the bundles ( with > the unpacked jars being loaded into a classloader ). > > If the unbundled is the same as the bundle, there is no op. In this case, > it is first run. > > So this is the parser bolt using the HDFS backed extension loader.” > > > This is part of the parser side loading, also feeds into the stellar load > from hdfs stuff. > > * metron has extensions > * extensions are packaged with their configurations and their ‘bundle’ ( > the nar part ) > * extension bundles are deployed to hdfs *lib directory > * parts of the system that use extensions discover and load them, with > class loaders setup for the bundle > * part of that, the bundles are unpackaged into a ‘working’ area if > required ( so if you drop a new version into the lib, the next app will > unpack the new version ) > * thus, the bundle system needs to be able to write to the working areas in > hdfs > * ??? > * profit > > > On April 12, 2017 at 08:11:28, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > I’m curious: Otto, what’s your use case for writing to HDFS from a parsers? > > Simon > > > On 12 Apr 2017, at 13:04, Justin Leet <justinjl...@gmail.com> wrote: > > > > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group > and > > with 775 will be able to write. > > > > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dlyle65...@gmail.com> > wrote: > > > >> It's curious to me that you're writing directly from parsing, but I > suspect > >> that your parsing topology is running as the storm user and it can't > write > >> to those directories. > >> > >> -D... > >> > >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ottobackwa...@gmail.com> > >> wrote: > >> > >>> The indexing dir is created: > >>> > >>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir, > >>> type="directory", > >>> action="create_on_execute", > >>> owner=self.__params.metron_user, > >>> group=self.__params.metron_group, > >>> mode=0775, > >>> ) > >>> > >>> > >>> > >>> > >>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwa...@gmail.com) > >>> wrote: > >>> > >>> > >>> I am trying to write to HDFS from ParserBolt, but I’m getting the > >> following > >>> exception: > >>> > >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied: > >>> user=storm, access=WRITE, > >>> inode="/apps/metron/extension_working/framework":metron: > hdfs:drwxrwxr-x > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. > >>> check(FSPermissionChecker.java:319) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. > >>> check(FSPermissionChecker.java:292) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. > >>> checkPermission(FSPermissionChecker.java:213) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. > >>> checkPermission(FSPermissionChecker.java:190) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. > >>> checkPermission(FSDirectory.java:1827) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. > >>> checkPermission(FSDirectory.java:1811) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. > checkAncestorAccess( > >>> FSDirectory.java:1794) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs( > >>> FSDirMkdirOp.java:71) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs( > >>> FSNamesystem.java:4011) > >>> at > >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. > >>> mkdirs(NameNodeRpcServer.java:1102) > >>> at > >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi > >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi > >>> deTranslatorPB.java:630) > >>> at > >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ > >>> ClientNamenodeProtocol$2.callBlockingMethod( > >> ClientNamenodeProtocolProtos. > >>> java) > >>> at > >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ > ProtoBufRpcInvoker.call( > >>> ProtobufRpcEngine.java:640) > >>> > >>> > >>> The HDFS directory is created as such: > >>> > >>> self.__params.HdfsResource(self.__params.hdfs_metron_ > >>> apps_extensions_working, > >>> type="directory", > >>> action="create_on_execute", > >>> owner=self.__params.metron_user, > >>> mode=0775) > >>> > >>> > >>> As the hdfs write handlers I am logging in as such: > >>> > >>> HdfsSecurityUtil.login(stormConfig, fsConf); > >>> FileSystem fileSystem = FileSystem.get(fsConf); > >>> > >>> I am not sure what is different from the indexing hdfs writer setup > here, > >>> but what I’m doing obviously is not working. > >>> > >>> Any ideas? > >>> > >>> > >>> - the branch: > >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy > >>> > >>> I am not up to date with master. > >>> > >> >
