abdullah alamoudi has submitted this change and it was merged. Change subject: Improve Error Handling in Local Directory Feeds ......................................................................
Improve Error Handling in Local Directory Feeds This change improves handling of two error types for filesystem based feeds. The first one is the handling of IO Errors which causes the input stream to be closed, and the second one is reacting to missed filesystem events. In both cases, we scan the directory and compare it with the history we have in order to resume from where we last left off. In addition, this change includes some refactoring in external data. Particularly, we get rid of the stream provider layer and instead, stream factories create input streams directly. This is consistent with record reader factories which create readers directly without reader providers. Change-Id: I08d89229e33c91532b1038ba9f7a372f7ca1fdb5 Reviewed-on: https://asterix-gerrit.ics.uci.edu/720 Tested-by: Jenkins <[email protected]> Reviewed-by: Murtadha Hubail <[email protected]> --- M .gitattributes M .gitignore M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java M asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java M asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java M asterix-app/src/test/resources/runtimets/queries/feeds/feeds_08/feeds_08.1.ddl.aql M asterix-app/src/test/resources/runtimets/queries/feeds/feeds_08/feeds_08.2.update.aql M asterix-app/src/test/resources/runtimets/queries/feeds/feeds_08/feeds_08.3.query.aql A asterix-app/src/test/resources/runtimets/queries/feeds/feeds_08/feeds_08.4.sleep.aql M asterix-app/src/test/resources/runtimets/results/external-library/classad-parser/classad-parser.1.adm M asterix-app/src/test/resources/runtimets/results/feeds/feed-with-external-parser/feed-with-external-parser.1.adm M asterix-app/src/test/resources/runtimets/testsuite.xml M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java R asterix-external-data/src/main/java/org/apache/asterix/external/api/AsterixInputStream.java M asterix-external-data/src/main/java/org/apache/asterix/external/api/IExternalIndexer.java M asterix-external-data/src/main/java/org/apache/asterix/external/api/IIndexingDatasource.java R asterix-external-data/src/main/java/org/apache/asterix/external/api/IInputStreamFactory.java M asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedRecordDataFlowController.java M asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedStreamDataFlowController.java M asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedTupleForwarder.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IFeedFrameHandler.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IFeedJoint.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IFeedLifecycleListener.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IMessageReceiver.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/DistributeFeedFrameWriter.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FeedCollectRuntimeInputHandler.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FeedFrameHandlers.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FeedRuntimeInputHandler.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FrameDistributor.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/management/FeedConnectionRequest.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/message/MessageReceiver.java M asterix-external-data/src/main/java/org/apache/asterix/external/feed/runtime/IngestionRuntime.java M asterix-external-data/src/main/java/org/apache/asterix/external/indexing/FileOffsetIndexer.java M asterix-external-data/src/main/java/org/apache/asterix/external/indexing/RecordColumnarIndexer.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/CharArrayRecord.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/converter/DCPRequestToRecordWithMetadataAndPKConverter.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/TestAsterixMembersReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/AbstractCharRecordLookupReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/HDFSRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/kv/KVReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/kv/KVTestReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/rss/RSSRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/AbstractStreamRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/AbstractStreamRecordReaderFactory.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/EmptyLineSeparatedRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/EmptyLineSeparatedRecordReaderFactory.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReaderFactory.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/QuotedLineRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReaderFactory.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/AsterixInputStreamReader.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/BasicInputStream.java A asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/HDFSInputStream.java R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/LocalFSInputStream.java A asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/SocketClientInputStream.java M asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/SocketServerInputStream.java A asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/TwitterFirehoseInputStream.java R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/factory/LocalFSInputStreamFactory.java R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/factory/SocketClientInputStreamFactory.java R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/factory/SocketServerInputStreamFactory.java R asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/factory/TwitterFirehoseStreamFactory.java D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/HDFSInputStreamProvider.java D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/LocalFSInputStreamProvider.java D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/SocketClientInputStreamProvider.java D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/SocketServerInputStreamProvider.java D asterix-external-data/src/main/java/org/apache/asterix/external/input/stream/provider/TwitterFirehoseInputStreamProvider.java M asterix-external-data/src/main/java/org/apache/asterix/external/operators/FeedCollectOperatorDescriptor.java M asterix-external-data/src/main/java/org/apache/asterix/external/operators/FeedCollectOperatorNodePushable.java M asterix-external-data/src/main/java/org/apache/asterix/external/parser/AbstractDataParser.java M asterix-external-data/src/main/java/org/apache/asterix/external/provider/DataflowControllerProvider.java M asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java M asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java M asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java M asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java M asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/AttributeReference.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/BuiltinClassAdFunctions.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/CaseInsensitiveString.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ClassAd.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ClassAdFunc.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ClassAdTime.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ClassAdUnParser.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/EvalState.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ExprList.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ExprTree.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/ExprTreeHolder.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/FunctionCall.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/Lexer.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/Literal.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/Operation.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/PrettyPrint.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/Value.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/AMutableCharArrayStringPool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/AttributeReferencePool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/CaseInsensitiveStringPool.java A asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ClassAdObjectPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ClassAdParserPool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ClassAdPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ClassAdTimePool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/DoublePool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/EvalStatePool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ExprHolderPool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ExprListPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/FunctionCallPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/HashMapPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/Int32Pool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/Int64Pool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/LiteralPool.java R asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/MutableBooleanPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/MutableNumberFactorPool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/OperationPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/PrettyPrintPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/StringArrayListPool.java C asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/TreeSetPool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/object/pool/ValuePool.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdFunctionalTest.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdParserTest.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdToADMTest.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdUnitTest.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdUnitTester.java M asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/FunctionalTester.java M asterix-external-data/src/test/java/org/apache/asterix/external/library/ClassAdParser.java M asterix-external-data/src/test/java/org/apache/asterix/external/library/ClassAdParserFactory.java M asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/RecordWithMetaTest.java M asterix-external-data/src/test/resources/functional_tests.txt M asterix-installer/src/test/resources/integrationts/library/results/library-parsers/record-parser/record-parser.1.adm M asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AqlMetadataProvider.java M asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/FeedDataSource.java M asterix-om/src/main/java/org/apache/asterix/dataflow/data/nontagged/serde/AStringSerializerDeserializer.java M asterix-om/src/main/java/org/apache/asterix/formats/nontagged/AqlSerializerDeserializerProvider.java M asterix-om/src/test/java/org/apache/asterix/dataflow/data/nontagged/serde/SimpleSerializerDeserializerTest.java 136 files changed, 3,333 insertions(+), 2,851 deletions(-) Approvals: Murtadha Hubail: Looks good to me, approved Jenkins: Verified -- To view, visit https://asterix-gerrit.ics.uci.edu/720 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: merged Gerrit-Change-Id: I08d89229e33c91532b1038ba9f7a372f7ca1fdb5 Gerrit-PatchSet: 15 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi <[email protected]> Gerrit-Reviewer: Ian Maxon <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Murtadha Hubail <[email protected]> Gerrit-Reviewer: Till Westmann <[email protected]> Gerrit-Reviewer: abdullah alamoudi <[email protected]>
