writer-jill commented on code in PR #12981: URL: https://github.com/apache/druid/pull/12981#discussion_r979973884
########## docs/ingestion/migrate-from-firehose-ingestion.md: ########## @@ -0,0 +1,196 @@ +--- +id: migrate-from-firehose +title: "Migrate from firehose to input source ingestion" +sidebar_label: "Migrate from firehose" +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +Apache deprecated support for Druid firehoses in version 0.17. Support for firehose ingestion was removed in LTS version 0.24. + +If you're using a firehose for batch ingestion, we strongly recommend that you follow the instructions on this page to transition to using native batch ingestion input sources as soon as possible. + +Firehose ingestion doesn't work with newer Druid versions, so you must be using an ingestion spec with a defined input source before you upgrade. + +## Migrate from firehose ingestion to an input source + +To migrate from firehose ingestion, copy your existing ingestion spec into a new file. Refer to [Native batch ingestion with firehose (Deprecated)](./native-batch-firehose.md) for a description of firehose properties. + +Edit the new file as follows: + +1. In the `ioConfig` component, replace the `firehose` definition with an `inputSource` definition for your chosen input source. See [Native batch input sources](./native-batch-input-sources) for details. +2. Move the `timeStampSpec` definition from `parser.parseSpec` to the `dataSchema` component. +3. Move the `dimensionsSpec` definition from `parser.parseSpec` to the `dataSchema` component. +4. Move the `format` definition from `parser.parseSpec` to an `inputFormat` definition in `ioConfig`. +5. Delete the `parser` definition. +6. Save the file. +<br>You can check the format of your new ingestion file against the [migrated example](#example-ingestion-spec-after-migration) below. +7. Test the new ingestion spec with a temporary data source. +8. Once you've successfully ingested sample data with the new spec, stop firehose ingestion and switch to the new spec. + +When the transition is complete, you can upgrade Druid to the latest version. See the [Druid release notes](https://druid.apache.org/downloads.html) for upgrade instructions. + +### Example firehose ingestion spec + +An example firehose ingestion spec is as follows: + +```json +{ + "type" : "index", + "spec" : { + "dataSchema" : { + "dataSource" : "wikipedia", + "metricsSpec" : [ + { + "type" : "count", + "name" : "count" + }, + { + "type" : "doubleSum", + "name" : "added", + "fieldName" : "added" + }, + { + "type" : "doubleSum", + "name" : "deleted", + "fieldName" : "deleted" + }, + { + "type" : "doubleSum", + "name" : "delta", + "fieldName" : "delta" + } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "DAY", + "queryGranularity" : "NONE", + "intervals" : [ "2013-08-31/2013-09-01" ] + }, + "parser": { Review Comment: @vogievetsky @tejaswini-imply Is there a reason why we can't replace these draft instructions with instructions using the web console? Would anyone need to perform the manual alternative? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
