----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/#review52324 -----------------------------------------------------------
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh <https://reviews.apache.org/r/22791/#comment91085> Mike, instead of directly including cluster-tools in oodt, can you simply make: # A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT. http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml <https://reviews.apache.org/r/22791/#comment91086> Custom Maven repos are difficult since the Central repository is phasing them out: http://blog.sonatype.com/2010/03/why-external-repos-are-being-phased-out-of-central/ If we absolutely need to ref this repo, can you make it a Maven profile, not enabled by default, so that users won't have the issue for this when downloading and building OODT? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java <https://reviews.apache.org/r/22791/#comment91087> how about instead of NA, for the filename, we call it a "stream-UUID" where we generate a unique stream UUID as the FILENAME field. http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java <https://reviews.apache.org/r/22791/#comment91088> how about a similarly derived STREAM loc here, perhaps the same as the FILENAME suggestion using UUID? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java <https://reviews.apache.org/r/22791/#comment91089> rather than silently do something, maybe take this out until there is something to do? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java <https://reviews.apache.org/r/22791/#comment91090> not sure what this change is? http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java <https://reviews.apache.org/r/22791/#comment91091> All of these files need ALv2 license headers. http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java <https://reviews.apache.org/r/22791/#comment91092> We shoudln't be checking in generated files - can this simply be generated during the Resmgr build? http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml <https://reviews.apache.org/r/22791/#comment91093> if we don't need this, remove it. http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak <https://reviews.apache.org/r/22791/#comment91094> ALv2 headers here and everywhere http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties <https://reviews.apache.org/r/22791/#comment91095> remove JPL specific URLs. - Chris Mattmann On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22791/ > ----------------------------------------------------------- > > (Updated Sept. 4, 2014, 5:23 p.m.) > > > Review request for oodt, Lewis McGibbney and Chris Mattmann. > > > Repository: oodt > > > Description > ------- > > This patch contains all the changes needed to add in "streaming oodt" into > the oodt svn repository. > > There are four main portions: > -Mesos Framework for Resource Manager (Prototype working) > -Spark Runner for Workflow Manager (Prototype working) > -Filemanager "streaming" type (In development) > -Deployment and cluster management scripts (In development) > > Where can this stuff be put so that it is available to use, even while it is > in development? > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java > 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties > PRE-CREATION > > Diff: https://reviews.apache.org/r/22791/diff/ > > > Testing > ------- > > Basic functionality tests done for both the resource-manger and workflow > manager pieces. Filemanager have been tested to properly ingest a > "GenericStream" type with the lucene catalog only. > > > Thanks, > > Michael Starch > >
