RKemper added a comment.
[data-transfer cookbook -> kafka offset output]
-----------------------------------------------
(including this output since it's the first time we've relied on the new
kafka offset part of the data-transfer process; won't be posting the output for
any subsequent cookbooks runs)
Also note that the output is a little tricky to parse. I can't tell if it did
succeed nad the `Error sending OffsetCommitRequest_v2...` log lines were errors
that were resolved upon retry, or if the cookbook falsely reported the offset.
But judging by the fact that `wdqs2005` is reporting the correct number of
triples, I have to assume that it worked properly. (This also lines up with the
fact that we do have log lines that seem to report success: `Offsets set for
target cluster "main", site "codfw" and consumer group "wdqs2005"`)
Transferring Kafka offsets
<BrokerConnection node_id=bootstrap-2 host=kafka-main2004.codfw.wmnet:9093
<connecting> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to
kafka-main2004.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:38', 9093, 0, 0)
IPv6]
Probing node bootstrap-2 broker version
<BrokerConnection node_id=bootstrap-2 host=kafka-main2004.codfw.wmnet:9093
<handshake> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection
complete.
Broker version identifed as 1.0.0
Set configuration api_version=(1, 0, 0) to skip auto check_version requests
on startup
<BrokerConnection node_id=bootstrap-1 host=kafka-main2004.codfw.wmnet:9093
<connecting> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to
kafka-main2004.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:38', 9093, 0, 0)
IPv6]
Probing node bootstrap-1 broker version
<BrokerConnection node_id=bootstrap-1 host=kafka-main2004.codfw.wmnet:9093
<handshake> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection
complete.
Broker version identifed as 1.0.0
Set configuration api_version=(1, 0, 0) to skip auto check_version requests
on startup
Same cluster, setting offsets...
<BrokerConnection node_id=2004 host=kafka-main2004.codfw.wmnet:9093
<connecting> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to
kafka-main2004.codfw.wmnet:9093
[('2620:0:860:104:10:192:48:38', 9093, 0, 0) IPv6]
<BrokerConnection node_id=2001 host=kafka-main2001.codfw.wmnet:9093
<connecting> [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: connecting to
kafka-main2001.codfw.wmnet:9093 [('2620:0:860:101:10:192:0:17', 9093, 0, 0)
IPv6]
<BrokerConnection node_id=2004 host=kafka-main2004.codfw.wmnet:9093
<handshake> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection
complete.
<BrokerConnection node_id=bootstrap-2 host=kafka-main2004.codfw.wmnet:9093
<connected> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing
connection.
<BrokerConnection node_id=2005 host=kafka-main2005.codfw.wmnet:9093
<connecting> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: connecting to
kafka-main2005.codfw.wmnet:9093
[('2620:0:860:104:10:192:48:46', 9093, 0, 0) IPv6]
Group coordinator for wdqs2008 is BrokerMetadata(nodeId='coordinator-2003',
host='kafka-main2003.codfw.wmnet', port=9093, rack=None)
Discovered coordinator coordinator-2003 for group wdqs2008
<BrokerConnection node_id=2001 host=kafka-main2001.codfw.wmnet:9093
<handshake> [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: Connection
complete.
<BrokerConnection node_id=coordinator-2003
host=kafka-main2003.codfw.wmnet:9093 <connecting> [IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: connecting to
kafka-main2003.codfw.wmnet:9093 [('2620:0:860:103:10:192:32:136', 9093, 0, 0)
IPv6]
<BrokerConnection node_id=2005 host=kafka-main2005.codfw.wmnet:9093
<handshake> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Connection
complete.
<BrokerConnection node_id=coordinator-2003
host=kafka-main2003.codfw.wmnet:9093 <handshake> [IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Connection complete.
Extracted offsets from source cluster "main", site "codfw" and consumer
group "wdqs2008".
<BrokerConnection node_id=bootstrap-3 host=kafka-main2005.codfw.wmnet:9093
<connecting> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: connecting to
kafka-main2005.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:46', 9093, 0, 0)
IPv6]
Group coordinator for wdqs2005 is BrokerMetadata(nodeId='coordinator-2003',
host='kafka-main2003.codfw.wmnet', port=9093, rack=None)
Discovered coordinator coordinator-2003 for group wdqs2005
Error sending OffsetCommitRequest_v2 to node coordinator-2003
[NodeNotReadyError: coordinator-2003]
<BrokerConnection node_id=coordinator-2003
host=kafka-main2003.codfw.wmnet:9093 <connecting> [IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: connecting to
kafka-main2003.codfw.wmnet:9093 [('2620:0:860:103:10:192:32:136', 9093, 0, 0)
IPv6]
Error sending OffsetCommitRequest_v2 to node coordinator-2003
[NodeNotReadyError: coordinator-2003]
Error sending OffsetCommitRequest_v2 to node coordinator-2003
[NodeNotReadyError: coordinator-2003]
<BrokerConnection node_id=bootstrap-3 host=kafka-main2005.codfw.wmnet:9093
<handshake> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Connection
complete.
Error sending OffsetCommitRequest_v2 to node coordinator-2003
[NodeNotReadyError: coordinator-2003]
<BrokerConnection node_id=coordinator-2003
host=kafka-main2003.codfw.wmnet:9093 <handshake> [IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Connection complete.
<BrokerConnection node_id=bootstrap-1 host=kafka-main2004.codfw.wmnet:9093
<connected> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing
connection.
<BrokerConnection node_id=bootstrap-3 host=kafka-main2005.codfw.wmnet:9093
<connected> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Closing
connection.
Offsets set for target cluster "main", site "codfw" and consumer group
"wdqs2005".
<BrokerConnection node_id=coordinator-2003
host=kafka-main2003.codfw.wmnet:9093 <connected> [IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Closing connection.
<BrokerConnection node_id=2004 host=kafka-main2004.codfw.wmnet:9093
<connected> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing
connection.
<BrokerConnection node_id=2001 host=kafka-main2001.codfw.wmnet:9093
<connected> [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: Closing
connection.
<BrokerConnection node_id=2005 host=kafka-main2005.codfw.wmnet:9093
<connected> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Closing
connection.
<BrokerConnection node_id=coordinator-2003
host=kafka-main2003.codfw.wmnet:9093 <connected> [IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Closing connection.
Done.
Starting services [systemctl start wdqs-blazegraph && sleep 10 && systemctl
start wdqs-updater]
TASK DETAIL
https://phabricator.wikimedia.org/T288231
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, Lalamarie69,
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY,
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314,
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan,
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas,
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________
Wikidata-bugs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
