> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote: > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java, > > line 82 > > <https://reviews.apache.org/r/22791/diff/2/?file=659644#file659644line82> > > > > how about a similarly derived STREAM loc here, perhaps the same as the > > FILENAME suggestion using UUID?
See: https://reviews.apache.org/r/27172 > On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote: > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java, > > line 30 > > <https://reviews.apache.org/r/22791/diff/2/?file=659646#file659646line30> > > > > not sure what this change is? Added guard clause for ProductStructure, so users cannot end up with an unknown structure. All tests that are in violation of this have been fixed. See: https://reviews.apache.org/r/27172 > On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote: > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java, > > line 76 > > <https://reviews.apache.org/r/22791/diff/2/?file=659645#file659645line76> > > > > rather than silently do something, maybe take this out until there is > > something to do? Done see: https://reviews.apache.org/r/27172 - Michael ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/#review52324 ----------------------------------------------------------- On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22791/ > ----------------------------------------------------------- > > (Updated Oct. 24, 2014, 10:35 p.m.) > > > Review request for oodt, Lewis McGibbney and Chris Mattmann. > > > Repository: oodt > > > Description > ------- > > This patch contains all the changes needed to add in "streaming oodt" into > the oodt svn repository. > > There are four main portions: > -Mesos Framework for Resource Manager (Prototype working) > -Spark Runner for Workflow Manager (Prototype working) > -Filemanager "streaming" type (In development) > -Deployment and cluster management scripts (In development) > > Where can this stuff be put so that it is available to use, even while it is > in development? > > > Note: Filemanager work (and corrections here-in) have been moved to sub-patch > review: https://reviews.apache.org/r/27172 > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java > 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties > PRE-CREATION > > Diff: https://reviews.apache.org/r/22791/diff/ > > > Testing > ------- > > Basic functionality tests done for both the resource-manger and workflow > manager pieces. Filemanager have been tested to properly ingest a > "GenericStream" type with the lucene catalog only. > > > Thanks, > > Michael Starch > >
