Yes AVRO is supported. The copy options are essentially the arguments that can be used in running copy in sql. See https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#r_COPY_command_examples-copy-from-avro
Sent from my iPhone > On Oct 4, 2023, at 7:33 AM, Ross, Christopher > <[email protected]> wrote: > > Hello, > > We have a group that want to bring in AVRO files and ingest them into > Redshift tables. I know that there is a S3ToRedshiftOperator but the only > examples are using CSV. I wanted to know if AVRO can be used as a "copy > option".Thanks > > > transfer_s3_to_redshift = S3ToRedshiftOperator( > task_id="transfer_s3_to_redshift", > redshift_conn_id=conn_id_name, > s3_bucket=bucket_name, > s3_key=S3_KEY_2, > schema="PUBLIC", > table=REDSHIFT_TABLE, > copy_options=["csv"], > ) > > ________________________________ > From: [email protected] <[email protected]> > Sent: Wednesday, October 4, 2023 10:26 AM > To: Ross, Christopher <[email protected]> > Subject: [EXTERNAL] WELCOME to [email protected] > > Hi! This is the ezmlm program. I'm managing the > [email protected] mailing list. > > I'm working for my owner, who can be reached > at [email protected]. > > Acknowledgment: I have added the address > > [email protected] > > to the dev mailing list. > > Welcome to [email protected]! > > Please save this message so that you know the address you are > subscribed under, in case you later want to unsubscribe or change your > subscription address. > > > --- Administrative commands for the dev list --- > > I can handle administrative requests automatically. Please > do not send them to the list address! Instead, send > your message to the correct command address: > > To subscribe to the list, send a message to: > <[email protected]> > > To remove your address from the list, send a message to: > <[email protected]> > > Send mail to the following for info and FAQ for this list: > <[email protected]> > <[email protected]> > > Similar addresses exist for the digest list: > <[email protected]> > <[email protected]> > > To get messages 123 through 145 (a maximum of 100 per request), mail: > <[email protected]> > > To get an index with subject and author for messages 123-456 , mail: > <[email protected]> > > They are always returned as sets of 100, max 2000 per request, > so you'll actually get 100-499. > > To receive all messages with the same subject as message 12345, > send a short message to: > <[email protected]> > > The messages should contain one line or word of text to avoid being > treated as sp@m, but I will ignore their content. > Only the ADDRESS you send to is important. > > You can start a subscription for an alternate address, > for example "[email protected]", just add a hyphen and your > address (with '=' instead of '@') after the command word: > <[email protected]> > > To stop subscription for this address, mail: > <[email protected]> > > In both cases, I'll send a confirmation message to that address. When > you receive it, simply reply to it to complete your subscription. > > If despite following these instructions, you do not get the > desired results, please contact my owner at > [email protected]. Please be patient, my owner is a > lot slower than I am ;-) > > --- Enclosed is a copy of the request I received. > > Return-Path: <[email protected]> > Received: (qmail 2935999 invoked by uid 116); 4 Oct 2023 14:26:42 -0000 > Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) > (95.217.134.168) > by apache.org (qpsmtpd/0.94) with ESMTP; Wed, 04 Oct 2023 14:26:42 +0000 > Authentication-Results: apache.org; auth=none > Received: from localhost (localhost [127.0.0.1]) > by spamproc1-he-fi.apache.org (ASF Mail Server at > spamproc1-he-fi.apache.org) with ESMTP id A7637C0A3B > for > <dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org>; > Wed, 4 Oct 2023 14:26:42 +0000 (UTC) > X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org > X-Spam-Flag: NO > X-Spam-Score: -0.002 > X-Spam-Level: > X-Spam-Status: No, score=-0.002 tagged_above=-999 required=6.31 > tests=[DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, > DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, > RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] > autolearn=disabled > Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); > dkim=pass (2048-bit key) header.d=redcross.org > Received: from mx1-he-de.apache.org ([116.203.227.195]) > by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) > (amavisd-new, port 10024) > with ESMTP id W1KnjWgs9yet > for > <dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org>; > Wed, 4 Oct 2023 14:26:41 +0000 (UTC) > Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=40.107.237.76; > helo=nam12-bn8-obe.outbound.protection.outlook.com; > [email protected]; receiver=<UNKNOWN> > Received: from NAM12-BN8-obe.outbound.protection.outlook.com > (mail-bn8nam12on2076.outbound.protection.outlook.com [40.107.237.76]) > by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with > ESMTPS id B55147F09C > for > <dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org>; > Wed, 4 Oct 2023 14:26:41 +0000 (UTC) > ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; > b=RxWI1Jj3Dt1LUGET8hzMAGcK2sGBrH23FRYHOKDfQ0F2f/kvJoBUKHmi1DbISdewnNdUWHtwzF3AmKapcjzE7hW++Ejoeo9xTcmRBfiYiHThQNi+E8jLdWiq5e0XYgNfcSIRyr0qxVFvoyU5n3/najz++CJy55Hzl+0Kxhm+6AdNaTr5thjS7JOD3JOkQLsBiO6r/vC7X2/FfZhrX/j0sCvRxpmXsRekIUhlzN0YFhDF4x3UWq2kyFZgXptclYz8/qjxsqBgU1kEmKNzp5D/os5wfJMH19ilJod2HG23ptQl1qDXx3iNTeGYaFelPR3V1FfJv7xH/peDZNeKaykGSQ== > ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; > s=arcselector9901; > h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; > bh=5On/NkjsorAgjQipp0nHcwQh3EqUDye6stT3pzjcfBg=; > b=Y+Tl/jVMj7KCYh/+JPOTao1mJmbszyKbNrDIr4XMLwx/NrqDk03feooFh2YF6PZDNXltNigZ3czdZhXXp9+VgiMzjNwBsdImUabZCFxUiQTl1ZbkKgpe1gLfjQiZeA474i6odW82qVanQxdotRXxwq9yw6tBVL3E8JUgyJ38RMqPRA4n6IK83PEih5+HLndK41TUgfjbeEvafVFuGTDTrwdnbvP1j0A5SMExar3F0Yidy2FqSOy5p56/wjTESDtEPBTNhQ1tTESRCsaXJF76wWsFjET2o2qkxZEqE5ulmMCHNtHuK7LScSrOgXpt9+7r6YjONphUsmoZg/hqCBxmVw== > ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass > smtp.mailfrom=redcross.org; dmarc=pass action=none header.from=redcross.org; > dkim=pass header.d=redcross.org; arc=none > DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redcross.org; > s=selector2; > h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; > bh=5On/NkjsorAgjQipp0nHcwQh3EqUDye6stT3pzjcfBg=; > b=umOiZEq1fztSG73+llz7YcgtkKaBqFUsi7vf7G0cqLCHlezaH0mh0W8E9NVGRRwpHyM5NVoub/AJ7L9Y6O67m4qs+7tX0labmzmgoftvma3rVW1UDq0acAk3X6nBPO2nsvcGStCQyL8ViXPe3SKfOck1ekWgwe8Q3IAfJxYY9AayiAehI5tKHinRl0CqGu1x4Rx0E9riEQFUu+OTOXXlwiTJhm8Y4G4O/KZoFeb1Uo0M4ACYfCqDbJlqBpswpPSpAAe1EBYPnogBjecbTm5j08ak9+D/LWJSSOsWz0RlNvo509xlOOwZoOITVqB3GnoRhtSog0t2MhXnCZK29k+hng== > Received: from SN6PR04MB4254.namprd04.prod.outlook.com (2603:10b6:805:33::32) > by DM6PR04MB6780.namprd04.prod.outlook.com (2603:10b6:5:1::19) with Microsoft > SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id > 15.20.6863.24; Wed, 4 Oct 2023 14:26:33 +0000 > Received: from SN6PR04MB4254.namprd04.prod.outlook.com > ([fe80::fcea:ef3b:4688:d6cd]) by SN6PR04MB4254.namprd04.prod.outlook.com > ([fe80::fcea:ef3b:4688:d6cd%5]) with mapi id 15.20.6838.016; Wed, 4 Oct 2023 > 14:26:33 +0000 > From: "Ross, Christopher" <[email protected]> > To: > "dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org" > > <dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org> > Subject: Dev List > Thread-Topic: Dev List > Thread-Index: AQHZ9s7DUB2ZymEUj0GVDBU50iT7eA== > Date: Wed, 4 Oct 2023 14:26:33 +0000 > Message-ID: > <sn6pr04mb42540ca7cfdbc0124127b3a4d5...@sn6pr04mb4254.namprd04.prod.outlook.com> > Accept-Language: en-US > Content-Language: en-US > X-MS-Has-Attach: > X-MS-TNEF-Correlator: > msip_labels: > authentication-results: dkim=none (message not signed) > header.d=none;dmarc=none action=none header.from=redcross.org; > x-ms-publictraffictype: Email > x-ms-traffictypediagnostic: SN6PR04MB4254:EE_|DM6PR04MB6780:EE_ > x-ms-office365-filtering-correlation-id: ad7db635-1c1c-4632-9c2c-08dbc4e5e897 > x-ms-exchange-atpmessageproperties: SA > x-ms-exchange-senderadcheck: 1 > x-ms-exchange-antispam-relay: 0 > x-microsoft-antispam: BCL:0; > x-microsoft-antispam-message-info: > lGMEHZEHy9ypgHOJgavMQDUrt1BrYBYDZJRrr1CGTH/srtAPE576Ii6S8tv+T2KNBXZTADUKPvyjeuxhOT/yAM1pYtHwhcn29sNf4qKCsxPN1lVbMvZ8PyPaZEwNsl4EDKOGBRmTpR6L2y8PyOg8iKw2ybTMlLm1jH57hPFjuULz+bNa5mZp4+p+h9lxCPxy17xR+3sJFu/qaHCA/27ceN62p910y3S1mYzm/WNAmx0yiw/EW0Zb2HGq6kyGMNjcjfa17M7reCxBT3VoEhykmk/6Ht+lMdZ+M91RRBkyEnIvhA9LZ18/oyna4R+NH6CsxZjzPXakmZds02V5ejJH0jWVtz66sK9Z66w9Em10FU1XX0fOaBjuHLmaQ5xI1DufJ82iPUW5sgWcnvr+DmaZqf4Z5ZaQCD8Nv/tx7MViI3KvILlZPHjRanlqPwEcau56AQbmJCSBM23PH1aChY7/loH717OnIWbcHz4cqQNqaDg26MAVuny2bGHsBotDtJja6Tj4r78iuHIpQXydVCphLOh2Hk8YGQeLFwLyKoXzmTmd+m6Kcg/NjY2H0/02nKe41nIulZTE37AeQm2F4J6ANXLkneCHkeDI4P5a3kkjQ+z/zpjJruFSFSlcHORwKtuW > x-forefront-antispam-report: > CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR04MB4254.namprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(346002)(376002)(186009)(1800799009)(451199024)(64100799003)(122000001)(7116003)(558084003)(5660300002)(26005)(52536014)(71200400001)(8936002)(8676002)(478600001)(86362001)(19627405001)(66556008)(66946007)(55016003)(66446008)(66476007)(64756008)(316002)(38100700002)(91956017)(4270600006)(6506007)(3480700007)(33656002)(38070700005)(76116006)(9686003)(7696005)(2906002)(41300700001);DIR:OUT;SFP:1101; > x-ms-exchange-antispam-messagedata-chunkcount: 1 > x-ms-exchange-antispam-messagedata-0: > =?iso-8859-1?Q?JTBMICe0F7hqrdx0wRXX1vKTFB+z1HRGg/0Jz9QMchPopKsFwgtchWBXWp?= > =?iso-8859-1?Q?53y/USPGRoaI5vK4xAcaC5wsT0kYW0EayeqG5urEFBz/NVPnyMVsd+kx0y?= > =?iso-8859-1?Q?yrcnuLticOXL+BV79ETINZ5C0dIZ2vO/CyL9HLpoqezaOMwrAXIQJdOSap?= > =?iso-8859-1?Q?lRIsKawKWL9CNvIpjjnvpbKzBmfv8JPqZHlVCi2wGGskEeyeKvPhvyc7+X?= > =?iso-8859-1?Q?euVnmC+VP01SFMejAtWP9oEmB2FUSAZHq7zb8rg0xE1B1oMEZwyXfb/PMw?= > =?iso-8859-1?Q?sLsPishZNGf8E0tfNgJ872Anr+udHCR5lfocBLVzTFxPfM1mXQFj59a31W?= > =?iso-8859-1?Q?o0s4TSzM+p47kIkg7Us38ZoqQfFHiNKZYIiPrOlRRy/rxPiw7zdxNUWNxC?= > =?iso-8859-1?Q?yUPEvGWVIla26CZHJM6fT72w4JOyT5U8GG1N5kSp7W9a4hNfEOs3010eW6?= > =?iso-8859-1?Q?VBaxPPlqQHFTBKmyBlQZvk7RIimMsS64zOp0GWc63fD1rlhVGFr/b0albe?= > =?iso-8859-1?Q?QIICLLdkL2KG1Af/EfPO3QLWr0gpIpDMGwOp36XftDW/35nYLkFukmlH+X?= > =?iso-8859-1?Q?6ppiFspjo/2q50R/9+2gy3+0xABr+fpBHh0nNPvb8R9WXqio2cxWhD0sHr?= > =?iso-8859-1?Q?LN0nQMo2tCIFH0vjNxqBELeyNFJHJy30iVcpBNLlYkDiTmAXuEh+fjvzo4?= > =?iso-8859-1?Q?4a9Rw9BuaIjkiNy7HIBc4HRp+Eid1Jh2Kb1Hmg9aTbnZzWy3nRZ7H7xzNb?= > =?iso-8859-1?Q?f/puHMei9Rbhv6/JZWvqVWhjCl9whnOGrEvMCYT35Toc2majbAxRWVKOUV?= > =?iso-8859-1?Q?xkgzSMTcGFPtO+vrPzRcouJskCr2wkPeZ4QZzCGZjHUpeRoNvJl0ztAjDx?= > =?iso-8859-1?Q?Uhorff1fmRHNvMaiihVj/uOk+NBV+gR2zAO3PL4LH/wzkbTgpV4l6palZR?= > =?iso-8859-1?Q?xi1d6SK57EPIyTfwTycx6O1+8RieOQvQzR/lU1qZarsfuO0g7hIksiB0c6?= > =?iso-8859-1?Q?xf+HKdwUth9YnfGGcquCCHhIr5unylc0sl1reuQXsXbDO/I+ICn6+1WMyV?= > =?iso-8859-1?Q?yvOISUtli4e0NahCVQ8H67oiLlrdS+NG9xTTUBzfvUe7VhVBm8xgLPDN3H?= > =?iso-8859-1?Q?C8tzupUiMEpsdui1Qedw7lmV8YVJhvmOfLujb2NnlvrEvVKZvXXWfydONh?= > =?iso-8859-1?Q?ScK27yqpEMEEJliP+j8kej/i+mXDM4yNfkVLTVSgpYPsDmGimERzAxaLlp?= > =?iso-8859-1?Q?g3HgzOBAX8C67Wnpej5zNbEF5IJZ6pvNv8mmJlUA25EE2/Y5P2n5w2da6Z?= > =?iso-8859-1?Q?5WHs8KNGtfDs6t28VlP98aQBPWl3KlfwQ52MjGDqwDBS1CDMRzpon/VB4e?= > =?iso-8859-1?Q?4t3H9gRNzCTdkB1xDZE6cj6JCpgWhkWzHRvq8DBobQqGeuNSiI/RFAvzmv?= > =?iso-8859-1?Q?Fpzus5jGDqQ29RN9BIMvbLxZB1LkvKd9/8h3oaOW1LbDxAApfNAnrL5llt?= > =?iso-8859-1?Q?72JrSfYHaCwSMWzqHiiRNglgzA82Hkv99ZxuJpkqQUMesHqNA00xcs/G0Y?= > =?iso-8859-1?Q?3BuD6jxWwUnKCYqHSRi3m5t1P5Z3fEHlWsq8b0ldLE6V9EUXwS6sPDokks?= > =?iso-8859-1?Q?B3/QIISgeGexFqar8mSkqDbcPRs14kFoxErOHFYNYqcQgBpKx516RBJw?= > =?iso-8859-1?Q?=3D=3D?= > Content-Type: multipart/alternative; > > boundary="_000_SN6PR04MB42540CA7CFDBC0124127B3A4D5CBASN6PR04MB4254namp_" > MIME-Version: 1.0 > X-OriginatorOrg: redcross.org > X-MS-Exchange-CrossTenant-AuthAs: Internal > X-MS-Exchange-CrossTenant-AuthSource: SN6PR04MB4254.namprd04.prod.outlook.com > X-MS-Exchange-CrossTenant-Network-Message-Id: > ad7db635-1c1c-4632-9c2c-08dbc4e5e897 > X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Oct 2023 14:26:33.1571 > (UTC) > X-MS-Exchange-CrossTenant-fromentityheader: Hosted > X-MS-Exchange-CrossTenant-id: dd5b5d42-c0d3-4ad2-b5f1-60edb3af2771 > X-MS-Exchange-CrossTenant-mailboxtype: HOSTED > X-MS-Exchange-CrossTenant-userprincipalname: > HscsH3/xQRsbCSq7qE82+bIwrUpT7dri4ozHJ80BNfEzYuIqCB4WELOPL3IcSY10x0WwZ/sWaprulkSIFCMqWQc+k68GtJnSH/qbeus6ZeA= > X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR04MB6780 > > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe.
