Great! By the way, I've opened FLUME-734 about this issue, it has some more log data there.
Thanks. -eran On Wed, Aug 10, 2011 at 16:55, Jonathan Hsieh <[email protected]> wrote: > Eran, > > I've dug into this and I think I've found the root cause of this bug. I'll > try to patch this today. > > Jon. > > > On Mon, Aug 8, 2011 at 7:41 AM, Eran Kutner <[email protected]> wrote: > >> Hi, >> I have 3 collectors running, all with this configuration: >> collectorSource(54001) | collector(600000) { >> escapedFormatDfs("hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/", >> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec")) } >> >> Right now tall three are going crazy, opening multiple HDFS files per >> second. All failing with this error: "OutputFormat instance can only write >> to the same OutputStream". >> This configuration used to work fine, for a few days, but I've seen this >> wild behavior happen a few times before. Usually restarting the service was >> enough to stop it. Now that doesn't seem to help. >> >> The logs look like this: >> >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Creating HDFS file: >> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy.tmp >> 11/08/08 10:33:12 INFO compress.CodecPool: Got brand-new compressor >> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Opening >> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Creating HDFS file: >> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy.tmp >> 11/08/08 10:33:12 INFO debug.StubbornAppendSink: append failed on event >> 'WEB514 [INFO Sun Aug 07 09:51:43 EDT 2011] { City : } { CookieID : } { >> Country : US } { Date : (long)-721216654471344504 (string) '??????F #??' >> (double)-2.2850700827524182E260 } { ErrCode : 0 '' } { Event : 637534208 '&' >> } { GigyaUID : } { IP : 75.107.140.48 } { IPNum : >> (long)5434591513138954240 (string) 'Kk???0' (double)2.110829276777375E55 } >> { Info : } { PartnerData : } { PartnerData_hash : (long)0 (string) '' >> (double)0.0 } { PartnerID : (long)8768254487304339456 (string) 'y??? ' >> (double)1.3781410651737394E278 } { ProviderUID : } { Referrer : >> http://www.heavy.com/action/girls/2011/08/the-20-hottest-photos-of-jessica-jane-clement/} >> { ReqID : d3b9c9f27a1848919003e25268b90c34 } { RetryCount : 0 '' } { >> RowType : GSEvent } { Server : WEB514 } { SiteAccountID : } { >> SiteAccountSNID : 0 '' } { State : } { Status : 50331648 ' ' } { TotalTime >> : 0 '' } { Type : js } { UCID : tSUUKaA13PGc62n9R9xn4PbqBwHxfeeuV15smIXotZY= >> } { UUID : } { UserAge : 48 } { UserGender : 0 '' } { Val : 0 '' } { dbTime >> : 0 '' } { log : } { memcacheTime : 0 '' } { provider : 0 '' } { request : >> } { siteID : (long)8919959604146733056 (string) '{??? ' >> (double)1.9842753626727537E288 } { source : 1315925605 'None' } ' with >> error: OutputFormat instance can only write to the same OutputStream >> 11/08/08 10:33:12 INFO rolling.RollSink: closing RollSink >> 'escapedFormatDfs( "hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/", >> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec") )' >> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Closing >> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Closing HDFS file: >> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy.tmp >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: done writing raw file to hdfs >> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Closing >> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Closing HDFS file: >> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy.tmp >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: done writing raw file to hdfs >> 11/08/08 10:33:12 INFO rolling.RollSink: opening RollSink >> 'escapedFormatDfs( "hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/", >> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec") )' >> 11/08/08 10:33:12 INFO debug.InsistentOpenDecorator: Opened MaskDecorator >> on try 0 >> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Opening >> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Creating HDFS file: >> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy.tmp >> 11/08/08 10:33:12 INFO compress.CodecPool: Got brand-new compressor >> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Opening >> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Creating HDFS file: >> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy.tmp >> 11/08/08 10:33:12 INFO debug.StubbornAppendSink: append failed on event >> 'WEB204 [INFO Mon Aug 08 10:13:54 EDT 2011] { City : } { CookieID : } { >> Country : DE } { Date : (long)6591305623858826888 (string) '[y??????#??' >> (double)4.440113505901932E132 } { IP : 93.231.131.246 } { IPNum : >> (long)6766522057732653056 (string) ']??????' (double)2.2940434972655672E144 >> } { Referrer : } { RowType : GSLoad } { Server : WEB204 } { State : } { >> UCID : Jyg2gzYokIn3id2wDfv71Q== } { isHTTPS : 48 } { partnerID : >> (long)-7147478690450898944 (string) '??????' (double)-6.42864763300399E-170 >> } { siteID : (long)5554354511524397056 (string) 'M' >> (double)2.162938607572601E63 } ' with error: OutputFormat instance can only >> write to the same OutputStream >> 11/08/08 10:33:12 INFO rolling.RollSink: closing RollSink >> 'escapedFormatDfs( "hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/", >> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec") )' >> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Closing >> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Closing HDFS file: >> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy.tmp >> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: done writing raw file to hdfs >> >> >> >> >> >> -eran >> >> > > > -- > // Jonathan Hsieh (shay) > // Software Engineer, Cloudera > // [email protected] > > >
