[
https://issues.apache.org/jira/browse/CHUKWA-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699843#action_12699843
]
Eric Yang commented on CHUKWA-136:
----------------------------------
Let's assume the checkpoint file writes:
add
org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
chukwaAgent 21281 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log
38209
And next request adds:
add
org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
chukwaAgent 0 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 0
This would not be considered as duplicated streams from the patch above. We
may have to do duplicate detection at adaptor level because the parameters
passed to the adaptor are non-uniform. However, be careful of the
implementation. We don't want too much cost in doing the adaptor duplication
detection. Add Ari, Jerome to charm in.
> Duplicated adaptors
> -------------------
>
> Key: CHUKWA-136
> URL: https://issues.apache.org/jira/browse/CHUKWA-136
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: data collection
> Reporter: Cheng
> Assignee: Cheng
> Priority: Blocker
> Attachments: chukwa-136.patch
>
>
> I did a telnet list command, it shows
> telnet localhost 9093
> Trying 127.0.0.1...
> Connected to localhost (127.0.0.1).
> Escape character is '^]'.
> list
> 1)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> Iostat 0 /grid/0/chukwa/var/log/Iostat.log 145600
> 2)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> Top 0 /grid/0/chukwa/var/log/Top.log 1742264
> 3)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chunkQueue 9339 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log
> 36825
> 4)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaHttpSender 14777
> /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
> 5)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaAgent 12449 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log
> 38209
> 6)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chunkQueue 12061 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log
> 36825
> 7)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> Sar 0 /grid/0/chukwa/var/log/Sar.log 271400
> 8)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaAgent 12817 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log
> 38209
> 9)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaHttpSender 19221
> /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
> 10)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> Ps 0 /grid/0/chukwa/var/log/Ps.log 3332933
> 11)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaAgent 13185 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log
> 38209
> 12)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> Netstat 0 /grid/0/chukwa/var/log/Netstat.log 15537
> 13)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chunkQueue 11788 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log
> 36825
> 14)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chunkQueue 4767 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log
> 36825
> 15)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> Df 0 /grid/0/chukwa/var/log/Df.log 54540
> 17)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaAgent 9873 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log
> 38209
> 16)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaHttpSender 18887
> /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
> 19)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaAgent 0 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log
> 38209
> 18)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chunkQueue 12334 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log
> 36825
> 21)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaHttpSender 6077
> /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
> 20)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaHttpSender 18553
> /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
> 23)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chunkQueue 20207 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log
> 36825
> 22)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaAgent 21281 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log
> 38209
> 25)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> CC 0 /grid/0/chukwa/var/log/collector.log 1040900
> 24)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaHttpSender 31669
> /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
> chukwAgent is listed 6 times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
