Hello,
We have a group that want to bring in AVRO files and ingest them into Redshift
tables. I know that there is a S3ToRedshiftOperator but the only examples are
using CSV. I wanted to know if AVRO can be used as a "copy option".Thanks
transfer_s3_to_redshift = S3ToRedshiftOperator(
task_id="transfer_s3_to_redshift",
redshift_conn_id=conn_id_name,
s3_bucket=bucket_name,
s3_key=S3_KEY_2,
schema="PUBLIC",
table=REDSHIFT_TABLE,
copy_options=["csv"],
)
________________________________
From: [email protected] <[email protected]>
Sent: Wednesday, October 4, 2023 10:26 AM
To: Ross, Christopher <[email protected]>
Subject: [EXTERNAL] WELCOME to [email protected]
Hi! This is the ezmlm program. I'm managing the
[email protected] mailing list.
I'm working for my owner, who can be reached
at [email protected].
Acknowledgment: I have added the address
[email protected]
to the dev mailing list.
Welcome to [email protected]!
Please save this message so that you know the address you are
subscribed under, in case you later want to unsubscribe or change your
subscription address.
--- Administrative commands for the dev list ---
I can handle administrative requests automatically. Please
do not send them to the list address! Instead, send
your message to the correct command address:
To subscribe to the list, send a message to:
<[email protected]>
To remove your address from the list, send a message to:
<[email protected]>
Send mail to the following for info and FAQ for this list:
<[email protected]>
<[email protected]>
Similar addresses exist for the digest list:
<[email protected]>
<[email protected]>
To get messages 123 through 145 (a maximum of 100 per request), mail:
<[email protected]>
To get an index with subject and author for messages 123-456 , mail:
<[email protected]>
They are always returned as sets of 100, max 2000 per request,
so you'll actually get 100-499.
To receive all messages with the same subject as message 12345,
send a short message to:
<[email protected]>
The messages should contain one line or word of text to avoid being
treated as sp@m, but I will ignore their content.
Only the ADDRESS you send to is important.
You can start a subscription for an alternate address,
for example "[email protected]", just add a hyphen and your
address (with '=' instead of '@') after the command word:
<[email protected]>
To stop subscription for this address, mail:
<[email protected]>
In both cases, I'll send a confirmation message to that address. When
you receive it, simply reply to it to complete your subscription.
If despite following these instructions, you do not get the
desired results, please contact my owner at
[email protected]. Please be patient, my owner is a
lot slower than I am ;-)
--- Enclosed is a copy of the request I received.
Return-Path: <[email protected]>
Received: (qmail 2935999 invoked by uid 116); 4 Oct 2023 14:26:42 -0000
Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org)
(95.217.134.168)
by apache.org (qpsmtpd/0.94) with ESMTP; Wed, 04 Oct 2023 14:26:42 +0000
Authentication-Results: apache.org; auth=none
Received: from localhost (localhost [127.0.0.1])
by spamproc1-he-fi.apache.org (ASF Mail Server at
spamproc1-he-fi.apache.org) with ESMTP id A7637C0A3B
for
<dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org>;
Wed, 4 Oct 2023 14:26:42 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org
X-Spam-Flag: NO
X-Spam-Score: -0.002
X-Spam-Level:
X-Spam-Status: No, score=-0.002 tagged_above=-999 required=6.31
tests=[DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2,
RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001]
autolearn=disabled
Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new);
dkim=pass (2048-bit key) header.d=redcross.org
Received: from mx1-he-de.apache.org ([116.203.227.195])
by localhost (spamproc1-he-fi.apache.org [95.217.134.168])
(amavisd-new, port 10024)
with ESMTP id W1KnjWgs9yet
for
<dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org>;
Wed, 4 Oct 2023 14:26:41 +0000 (UTC)
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=40.107.237.76;
helo=nam12-bn8-obe.outbound.protection.outlook.com;
[email protected]; receiver=<UNKNOWN>
Received: from NAM12-BN8-obe.outbound.protection.outlook.com
(mail-bn8nam12on2076.outbound.protection.outlook.com [40.107.237.76])
by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with
ESMTPS id B55147F09C
for
<dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org>;
Wed, 4 Oct 2023 14:26:41 +0000 (UTC)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=RxWI1Jj3Dt1LUGET8hzMAGcK2sGBrH23FRYHOKDfQ0F2f/kvJoBUKHmi1DbISdewnNdUWHtwzF3AmKapcjzE7hW++Ejoeo9xTcmRBfiYiHThQNi+E8jLdWiq5e0XYgNfcSIRyr0qxVFvoyU5n3/najz++CJy55Hzl+0Kxhm+6AdNaTr5thjS7JOD3JOkQLsBiO6r/vC7X2/FfZhrX/j0sCvRxpmXsRekIUhlzN0YFhDF4x3UWq2kyFZgXptclYz8/qjxsqBgU1kEmKNzp5D/os5wfJMH19ilJod2HG23ptQl1qDXx3iNTeGYaFelPR3V1FfJv7xH/peDZNeKaykGSQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
bh=5On/NkjsorAgjQipp0nHcwQh3EqUDye6stT3pzjcfBg=;
b=Y+Tl/jVMj7KCYh/+JPOTao1mJmbszyKbNrDIr4XMLwx/NrqDk03feooFh2YF6PZDNXltNigZ3czdZhXXp9+VgiMzjNwBsdImUabZCFxUiQTl1ZbkKgpe1gLfjQiZeA474i6odW82qVanQxdotRXxwq9yw6tBVL3E8JUgyJ38RMqPRA4n6IK83PEih5+HLndK41TUgfjbeEvafVFuGTDTrwdnbvP1j0A5SMExar3F0Yidy2FqSOy5p56/wjTESDtEPBTNhQ1tTESRCsaXJF76wWsFjET2o2qkxZEqE5ulmMCHNtHuK7LScSrOgXpt9+7r6YjONphUsmoZg/hqCBxmVw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=redcross.org; dmarc=pass action=none header.from=redcross.org;
dkim=pass header.d=redcross.org; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redcross.org;
s=selector2;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=5On/NkjsorAgjQipp0nHcwQh3EqUDye6stT3pzjcfBg=;
b=umOiZEq1fztSG73+llz7YcgtkKaBqFUsi7vf7G0cqLCHlezaH0mh0W8E9NVGRRwpHyM5NVoub/AJ7L9Y6O67m4qs+7tX0labmzmgoftvma3rVW1UDq0acAk3X6nBPO2nsvcGStCQyL8ViXPe3SKfOck1ekWgwe8Q3IAfJxYY9AayiAehI5tKHinRl0CqGu1x4Rx0E9riEQFUu+OTOXXlwiTJhm8Y4G4O/KZoFeb1Uo0M4ACYfCqDbJlqBpswpPSpAAe1EBYPnogBjecbTm5j08ak9+D/LWJSSOsWz0RlNvo509xlOOwZoOITVqB3GnoRhtSog0t2MhXnCZK29k+hng==
Received: from SN6PR04MB4254.namprd04.prod.outlook.com (2603:10b6:805:33::32)
by DM6PR04MB6780.namprd04.prod.outlook.com (2603:10b6:5:1::19) with Microsoft
SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.6863.24; Wed, 4 Oct 2023 14:26:33 +0000
Received: from SN6PR04MB4254.namprd04.prod.outlook.com
([fe80::fcea:ef3b:4688:d6cd]) by SN6PR04MB4254.namprd04.prod.outlook.com
([fe80::fcea:ef3b:4688:d6cd%5]) with mapi id 15.20.6838.016; Wed, 4 Oct 2023
14:26:33 +0000
From: "Ross, Christopher" <[email protected]>
To:
"dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org"
<dev-sc.1696429492.bjopaecefhmlmdmfhpce-christopher.ross2=redcross....@airflow.apache.org>
Subject: Dev List
Thread-Topic: Dev List
Thread-Index: AQHZ9s7DUB2ZymEUj0GVDBU50iT7eA==
Date: Wed, 4 Oct 2023 14:26:33 +0000
Message-ID:
<sn6pr04mb42540ca7cfdbc0124127b3a4d5...@sn6pr04mb4254.namprd04.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
msip_labels:
authentication-results: dkim=none (message not signed)
header.d=none;dmarc=none action=none header.from=redcross.org;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: SN6PR04MB4254:EE_|DM6PR04MB6780:EE_
x-ms-office365-filtering-correlation-id: ad7db635-1c1c-4632-9c2c-08dbc4e5e897
x-ms-exchange-atpmessageproperties: SA
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:
lGMEHZEHy9ypgHOJgavMQDUrt1BrYBYDZJRrr1CGTH/srtAPE576Ii6S8tv+T2KNBXZTADUKPvyjeuxhOT/yAM1pYtHwhcn29sNf4qKCsxPN1lVbMvZ8PyPaZEwNsl4EDKOGBRmTpR6L2y8PyOg8iKw2ybTMlLm1jH57hPFjuULz+bNa5mZp4+p+h9lxCPxy17xR+3sJFu/qaHCA/27ceN62p910y3S1mYzm/WNAmx0yiw/EW0Zb2HGq6kyGMNjcjfa17M7reCxBT3VoEhykmk/6Ht+lMdZ+M91RRBkyEnIvhA9LZ18/oyna4R+NH6CsxZjzPXakmZds02V5ejJH0jWVtz66sK9Z66w9Em10FU1XX0fOaBjuHLmaQ5xI1DufJ82iPUW5sgWcnvr+DmaZqf4Z5ZaQCD8Nv/tx7MViI3KvILlZPHjRanlqPwEcau56AQbmJCSBM23PH1aChY7/loH717OnIWbcHz4cqQNqaDg26MAVuny2bGHsBotDtJja6Tj4r78iuHIpQXydVCphLOh2Hk8YGQeLFwLyKoXzmTmd+m6Kcg/NjY2H0/02nKe41nIulZTE37AeQm2F4J6ANXLkneCHkeDI4P5a3kkjQ+z/zpjJruFSFSlcHORwKtuW
x-forefront-antispam-report:
CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR04MB4254.namprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(136003)(366004)(346002)(376002)(186009)(1800799009)(451199024)(64100799003)(122000001)(7116003)(558084003)(5660300002)(26005)(52536014)(71200400001)(8936002)(8676002)(478600001)(86362001)(19627405001)(66556008)(66946007)(55016003)(66446008)(66476007)(64756008)(316002)(38100700002)(91956017)(4270600006)(6506007)(3480700007)(33656002)(38070700005)(76116006)(9686003)(7696005)(2906002)(41300700001);DIR:OUT;SFP:1101;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
=?iso-8859-1?Q?JTBMICe0F7hqrdx0wRXX1vKTFB+z1HRGg/0Jz9QMchPopKsFwgtchWBXWp?=
=?iso-8859-1?Q?53y/USPGRoaI5vK4xAcaC5wsT0kYW0EayeqG5urEFBz/NVPnyMVsd+kx0y?=
=?iso-8859-1?Q?yrcnuLticOXL+BV79ETINZ5C0dIZ2vO/CyL9HLpoqezaOMwrAXIQJdOSap?=
=?iso-8859-1?Q?lRIsKawKWL9CNvIpjjnvpbKzBmfv8JPqZHlVCi2wGGskEeyeKvPhvyc7+X?=
=?iso-8859-1?Q?euVnmC+VP01SFMejAtWP9oEmB2FUSAZHq7zb8rg0xE1B1oMEZwyXfb/PMw?=
=?iso-8859-1?Q?sLsPishZNGf8E0tfNgJ872Anr+udHCR5lfocBLVzTFxPfM1mXQFj59a31W?=
=?iso-8859-1?Q?o0s4TSzM+p47kIkg7Us38ZoqQfFHiNKZYIiPrOlRRy/rxPiw7zdxNUWNxC?=
=?iso-8859-1?Q?yUPEvGWVIla26CZHJM6fT72w4JOyT5U8GG1N5kSp7W9a4hNfEOs3010eW6?=
=?iso-8859-1?Q?VBaxPPlqQHFTBKmyBlQZvk7RIimMsS64zOp0GWc63fD1rlhVGFr/b0albe?=
=?iso-8859-1?Q?QIICLLdkL2KG1Af/EfPO3QLWr0gpIpDMGwOp36XftDW/35nYLkFukmlH+X?=
=?iso-8859-1?Q?6ppiFspjo/2q50R/9+2gy3+0xABr+fpBHh0nNPvb8R9WXqio2cxWhD0sHr?=
=?iso-8859-1?Q?LN0nQMo2tCIFH0vjNxqBELeyNFJHJy30iVcpBNLlYkDiTmAXuEh+fjvzo4?=
=?iso-8859-1?Q?4a9Rw9BuaIjkiNy7HIBc4HRp+Eid1Jh2Kb1Hmg9aTbnZzWy3nRZ7H7xzNb?=
=?iso-8859-1?Q?f/puHMei9Rbhv6/JZWvqVWhjCl9whnOGrEvMCYT35Toc2majbAxRWVKOUV?=
=?iso-8859-1?Q?xkgzSMTcGFPtO+vrPzRcouJskCr2wkPeZ4QZzCGZjHUpeRoNvJl0ztAjDx?=
=?iso-8859-1?Q?Uhorff1fmRHNvMaiihVj/uOk+NBV+gR2zAO3PL4LH/wzkbTgpV4l6palZR?=
=?iso-8859-1?Q?xi1d6SK57EPIyTfwTycx6O1+8RieOQvQzR/lU1qZarsfuO0g7hIksiB0c6?=
=?iso-8859-1?Q?xf+HKdwUth9YnfGGcquCCHhIr5unylc0sl1reuQXsXbDO/I+ICn6+1WMyV?=
=?iso-8859-1?Q?yvOISUtli4e0NahCVQ8H67oiLlrdS+NG9xTTUBzfvUe7VhVBm8xgLPDN3H?=
=?iso-8859-1?Q?C8tzupUiMEpsdui1Qedw7lmV8YVJhvmOfLujb2NnlvrEvVKZvXXWfydONh?=
=?iso-8859-1?Q?ScK27yqpEMEEJliP+j8kej/i+mXDM4yNfkVLTVSgpYPsDmGimERzAxaLlp?=
=?iso-8859-1?Q?g3HgzOBAX8C67Wnpej5zNbEF5IJZ6pvNv8mmJlUA25EE2/Y5P2n5w2da6Z?=
=?iso-8859-1?Q?5WHs8KNGtfDs6t28VlP98aQBPWl3KlfwQ52MjGDqwDBS1CDMRzpon/VB4e?=
=?iso-8859-1?Q?4t3H9gRNzCTdkB1xDZE6cj6JCpgWhkWzHRvq8DBobQqGeuNSiI/RFAvzmv?=
=?iso-8859-1?Q?Fpzus5jGDqQ29RN9BIMvbLxZB1LkvKd9/8h3oaOW1LbDxAApfNAnrL5llt?=
=?iso-8859-1?Q?72JrSfYHaCwSMWzqHiiRNglgzA82Hkv99ZxuJpkqQUMesHqNA00xcs/G0Y?=
=?iso-8859-1?Q?3BuD6jxWwUnKCYqHSRi3m5t1P5Z3fEHlWsq8b0ldLE6V9EUXwS6sPDokks?=
=?iso-8859-1?Q?B3/QIISgeGexFqar8mSkqDbcPRs14kFoxErOHFYNYqcQgBpKx516RBJw?=
=?iso-8859-1?Q?=3D=3D?=
Content-Type: multipart/alternative;
boundary="_000_SN6PR04MB42540CA7CFDBC0124127B3A4D5CBASN6PR04MB4254namp_"
MIME-Version: 1.0
X-OriginatorOrg: redcross.org
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN6PR04MB4254.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id:
ad7db635-1c1c-4632-9c2c-08dbc4e5e897
X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Oct 2023 14:26:33.1571
(UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: dd5b5d42-c0d3-4ad2-b5f1-60edb3af2771
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname:
HscsH3/xQRsbCSq7qE82+bIwrUpT7dri4ozHJ80BNfEzYuIqCB4WELOPL3IcSY10x0WwZ/sWaprulkSIFCMqWQc+k68GtJnSH/qbeus6ZeA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR04MB6780
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
