The channel serializes the flume event as avro including the headers, the http headers become event headers However the sink should only store the content, not the headers
On 2 March 2016 at 09:51, Baris Akgun (Garanti Teknoloji) < [email protected]> wrote: > No, we send json twitter data but in flume channel ı saw content type word > for each tweet. Is it normal ? How can ı send just tweets json without any > content type. I took tweets json from GNIP company. > > Thanks > iPhone'umdan gönderildi > > 2 Mar 2016 tarihinde 10:56 saatinde, Gonzalo Herreros <[email protected]> > şunları yazdı: > > Could it be that you are serializing avro instead of json? > > On 2 March 2016 at 08:25, Baris Akgun (Garanti Teknoloji) < > [email protected]> wrote: > >> Hi, >> >> >> >> When I send json data to flume with using http post, flume adds >> Co**ntent-Typeapplication/json** for each json post. >> >> >> >> In my http post java code, I give the content-type with using >> >> >> >> **con.setRequestProperty("Content-Type", "application/json");** function. >> >> >> >> >> >> I am using blob handler. >> >> >> >> **In flume conf file** >> >> >> >> *tier1.sources.source1.type = org.apache.flume.source.http.HTTPSource >> >> tier1.sources.source1.handler = >> org.apache.flume.sink.solr.morphline.BlobHandler* >> >> >> >> In flume channel, flume adds content type for each post as you see. After >> HDFS sink, The content type word causes a problem when ı try to parse json >> with spark sql or hive serDe. >> >> >> >> **The flume channel log data** >> >> >> >> *^LContent-Typeapplication/jsonú{"id":"+ag:_ea_ch.++i++e_.c- >> >> ^LContentTypeapplication/json‘{"id":"tag:search.twitter.com >> ,2005:642913165047648* >> >> >> >> Is there any idea for that problem? >> >> >> >> Thank a lot. >> >> >> >> *Barış Akgün* >> Analitik Veri Ambarı ve Büyük Veri Yönetimi >> Uzman >> >> Tel >> >> : >> >> Dahili >> >> : >> >> Faks >> >> : >> >> >> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir >> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis >> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune >> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde >> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da >> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin >> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, >> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve >> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. >> >> This message and attachments are confidential and intended solely for the >> individual(s) stated in this message. If you received this message although >> you are not the addressee, you are responsible to keep the message >> confidential. The sender has no responsibility for the accuracy or >> correctness of the information in the message and its attachments. Our >> company shall have no liability for any changes or late receiving, loss of >> integrity and confidentiality, viruses and any damages caused in anyway to >> your computer system. >> > > Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir > ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis > olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune > uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde > yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da > sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin > ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, > butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve > bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. > > This message and attachments are confidential and intended solely for the > individual(s) stated in this message. If you received this message although > you are not the addressee, you are responsible to keep the message > confidential. The sender has no responsibility for the accuracy or > correctness of the information in the message and its attachments. Our > company shall have no liability for any changes or late receiving, loss of > integrity and confidentiality, viruses and any damages caused in anyway to > your computer system. >
