Thanks, I will try it. בתאריך יום ב׳, 21 בספט׳ 2015, 18:15 מאת Ahmed Vila <[email protected]>:
> I think that interceptor is a way to go and you can use regex_extractor > interceptor instead of building your own - it simplifies deployment. > https://flume.apache.org/FlumeUserGuide.html#regex-extractor-interceptor > > Further, you can use RegexExtractorInterceptorMillisSerializer for this > interceptor's type in order to do datetime parsing and do the formatting on > fly that will be suitable for your hdfs folder (f.e. extract year-month). > https://flume.apache.org/FlumeUserGuide.html#example-2 > > Later on, you can use header replacement string in the hdfs path to inject > a header defined with serializer's name ("timestamp" in the example above). > Hdfs Sink supports timestamp header out of the box and applies escape > sequences against it by default, unless hdfs.useLocalTimeStamp = true. > So, a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S will have > replacement values from the timestamp header produced by regex extractor. > > > > > On Mon, Sep 21, 2015 at 5:03 PM, IT CTO <[email protected]> wrote: > >> I want to read files and write them to hdfs but I want to write the to a >> date partitioned folder based on a date value IN THE ROW. >> Should I write a custom interceptor or custom serializer? >> Eran >> -- >> Eran | "You don't need eyes to see, you need vision" (Faithless) >> > > > > -- > > Best regards, > Ahmed Vila | Senior software developer > DevLogic | Sarajevo | Bosnia and Herzegovina > > Office : +387 33 942 123 > Mobile: +387 62 139 348 > > Website: www.devlogic.eu > E-mail : [email protected] > --------------------------------------------------------------------- > This e-mail and any attachment is for authorised use by the intended > recipient(s) only. This email contains confidential information. It should > not be copied, disclosed to, retained or used by, any party other than the > intended recipient. Any unauthorised distribution, dissemination or copying > of this E-mail or its attachments, and/or any use of any information > contained in them, is strictly prohibited and may be illegal. If you are > not an intended recipient then please promptly delete this e-mail and any > attachment and all copies and inform the sender directly via email. Any > emails that you send to us may be monitored by systems or persons other > than the named communicant for the purposes of ascertaining whether the > communication complies with the law and company policies. > > --------------------------------------------------------------------- > This e-mail and any attachment is for authorised use by the intended > recipient(s) only. This email contains confidential information. It should > not be copied, disclosed to, retained or used by, any party other than the > intended recipient. Any unauthorised distribution, dissemination or copying > of this E-mail or its attachments, and/or any use of any information > contained in them, is strictly prohibited and may be illegal. If you are > not an intended recipient then please promptly delete this e-mail and any > attachment and all copies and inform the sender directly via email. Any > emails that you send to us may be monitored by systems or persons other > than the named communicant for the purposes of ascertaining whether the > communication complies with the law and company policies. -- Eran | "You don't need eyes to see, you need vision" (Faithless)
