[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139728#comment-13139728 ]
Mck SembWever edited comment on CASSANDRA-3150 at 10/30/11 7:46 PM: -------------------------------------------------------------------- Back after an upgrade to cassandra-1.0.0 Example job start logs{noformat}[ INFO] 20:39:21 Restricting input range: 3589a548d20f80a7b41368b59973bcbc -- 36f0bedaf02a49a3b41368b59973bcbc [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:222) [ INFO] 20:39:21 Corresponding time range is Sun Oct 30 00:00:00 CEST 2011 (1319925600000) -- Sun Oct 30 23:00:00 CET 2011 (1320012000000) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:225) [ INFO] 20:39:21 Starting AdIdCountAggregation-phase1-DAY_2011303 ( to=1320002999000)) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.run(GenericCountAggregation.java:142) [DEBUG] 20:39:21 adding ColumnFamilySplit{startToken='3589a548d20f80a7b41368b59973bcbc', endToken='36f0bedaf02a49a3b41368b59973bcbc', dataNodes=[0.0.0.0]} [] at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:210) {noformat} In this split there is in fact ~40 million rows, and with a split size of 393216, it is expected ~100 splits to be returned. I'm also very confused by the dataNodes=[0.0.0.0]. this looks to be wrong, i know that data lies on the third node: cassandra03, where-else the job is running on the first node: cassandra01. was (Author: michaelsembwever): Back after an upgrade to cassandra-1.0.0 Example job start logs{noformat}[ INFO] 20:39:21 Restricting input range: 3589a548d20f80a7b41368b59973bcbc -- 36f0bedaf02a49a3b41368b59973bcbc [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:222) [ INFO] 20:39:21 Corresponding time range is Sun Oct 30 00:00:00 CEST 2011 (1319925600000) -- Sun Oct 30 23:00:00 CET 2011 (1320012000000) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:225) [ INFO] 20:39:21 Starting AdIdCountAggregation-phase1-DAY_2011303 ( to=1320002999000)) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.run(GenericCountAggregation.java:142) [DEBUG] 20:39:21 adding ColumnFamilySplit{startToken='3589a548d20f80a7b41368b59973bcbc', endToken='36f0bedaf02a49a3b41368b59973bcbc', dataNodes=[0.0.0.0]} [] at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:210) {noformat} In this split there is in fact ~40 million rows, and with a split size of 393216, it is expected ~100 splits to be returned. I'm also very confused by the {{dataNodes=[0.0.0.0]}}. this looks to be wrong, i know that data lies on the third node: cassandra03, where-else the job is running on the first node: cassandra01. > ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of > whack) > ---------------------------------------------------------------------------------- > > Key: CASSANDRA-3150 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3150 > Project: Cassandra > Issue Type: Bug > Components: Hadoop > Affects Versions: 0.8.4, 0.8.5 > Reporter: Mck SembWever > Assignee: Mck SembWever > Priority: Critical > Fix For: 0.8.6 > > Attachments: CASSANDRA-3150.patch, Screenshot-Counters for > task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map > task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, > attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, > fullscan-example1.log > > > From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039 > {quote} > bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner > bq. CFIF's inputSplitSize=196608 > bq. 3 map tasks (from 4013) is still running after read 25 million rows. > bq. Can this be a bug in StorageService.getSplits(..) ? > getSplits looks pretty foolproof to me but I guess we'd need to add > more debug logging to rule out a bug there for sure. > I guess the main alternative would be a bug in the recordreader paging. > {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira