> On March 5, 2014, 6:33 p.m., Neha Narkhede wrote: > > core/src/main/scala/kafka/tools/newproducer/MirrorMaker.scala, line 159 > > <https://reviews.apache.org/r/18774/diff/1/?file=510875#file510875line159> > > > > I had initially added multiple producer support to the mirror maker > > until we are fully convinced that a single producer's throughput with the > > highest durability setting can match the consumer's throughput. I haven't > > seen us publish these performance results yet. So to allow configuring > > MirrorMaker with multiple producers, I suggest we keep it as it at least > > for a while > > Jun Rao wrote: > The issue is that the new MirrorMaker right now is in an inconsistent > state. The code has the logic to support multiple producers, yet there is no > command line option to allow that. I'd prefer to keep the code in a > consistent state. If we want add the # producers option later, it's not too > hard to add those code back. > > Neha Narkhede wrote: > Let's talk about this. I almost think adding/fixing the command line > option is easier than adding all this code back. > > Neha Narkhede wrote: > Jun, one improvement that I can think off is actually in error reporting. > Currently, the producer only reports n-1 failed attempts but it doesn't > report the last failed attempt when it drops the message on the floor. For > reporting this, we need to get the response from the send() and check for > exceptions, if any and if so, log an error. > > Let me know if you prefer to file a JIRA instead of fixing it here.
For the error log, that's what I added in ProducerPerformance. In sync mode, futureResponse.get() will throw an exception if the send fails and we log the exception. In async mode, it creates a callback to log any failed send. For MirrorMaker, I didn't realize that it actually supports multiple producers. We just use #streams as #producers. I will remove the changes in MirrorMaker then. - Jun ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18774/#review36245 ----------------------------------------------------------- On March 6, 2014, 5:36 p.m., Jun Rao wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18774/ > ----------------------------------------------------------- > > (Updated March 6, 2014, 5:36 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-1240 > https://issues.apache.org/jira/browse/KAFKA-1240 > > > Repository: kafka > > > Description > ------- > > Log error when running in async mode. > > > 1. Converted all existing testcases in replication_testsuite and > mirror_maker_testsuite to use the new producer. 2. Minor cleanup to > ProducerPerformance and MirrorMaker. 3. Fixed a few places in python script > where we forget to release the lock, which can cause the system test to hang > when stopping all entities. 4. Left migration_testsuite as it is. We probably > can just delete the suite when we phase out the old producer. 5. There are 4 > testcase failures in replication_testsuite and 2 testcase failsures in > mirror_maker. These may be fixed with the configrable retry backoff time. > Will rerun the tests. 6. To void having to duplicate too many test cases > exercising both the old and the new producer, I propose that we just > duplicate the following testcases to run with the old producer (1) all > testcases in mirrormaker_testsuite and (2) testcases 0101-0110 and 4011-4018 > in replication_testsuite. Most testcases in replication_testsuite are really > for testing the replication logic on the br okers. From the producer perspective, as long as we can exercise the leader failure case, it doesn't really matter how the leader fails. > > > Diffs > ----- > > core/src/main/scala/kafka/tools/newproducer/MirrorMaker.scala > d23ef9a31aa0cc38b60e7651ae85ad6ad7dfcbf4 > perf/src/main/scala/kafka/perf/ProducerPerformance.scala > 5d399d94eafad7e3e50c538810f0aff53c59073f > system_test/migration_tool_testsuite/migration_tool_test.py > ce6f4f684beaa272409f6e25b74c957a9591206b > system_test/mirror_maker_testsuite/config/mirror_producer.properties > b2bf2c23b6b397571113fba7e75cead6404561a4 > system_test/mirror_maker_testsuite/mirror_maker_test.py > dfffb4e977ef22f48c6e503687b89ac33edc5338 > > system_test/mirror_maker_testsuite/testcase_5001/testcase_5001_properties.json > 287cab9a9b6b9e9d117905a1ac06708a74dfadcd > > system_test/mirror_maker_testsuite/testcase_5002/testcase_5002_properties.json > 5457eb1d26526759dcdd69895be633f76c01506a > > system_test/mirror_maker_testsuite/testcase_5003/testcase_5003_properties.json > 98fefee5838a2c7a1255042ac637dae46d72f51c > > system_test/mirror_maker_testsuite/testcase_5004/testcase_5004_properties.json > 6067b1263d0a5ebad0bd2846da89262c79d72f26 > > system_test/mirror_maker_testsuite/testcase_5005/testcase_5005_properties.json > 58483ad375cd1c7ec8dd8586af142f1b9c1bf437 > > system_test/mirror_maker_testsuite/testcase_5006/testcase_5006_properties.json > 1d9190ce7c1b5eb72b43d669e777a6398c25eed6 > system_test/replication_testsuite/replica_basic_test.py > e20130b85aecdacc8473f3e16abba007a3eeb3eb > > system_test/replication_testsuite/testcase_0001/testcase_0001_properties.json > 9bc164bee1dd0ad64904598fed555c9229cf82a4 > > system_test/replication_testsuite/testcase_0002/testcase_0002_properties.json > c90d7536a75641322562a018dd089081d5f5788f > > system_test/replication_testsuite/testcase_0003/testcase_0003_properties.json > b62b8aa50a38cb20906781815ca050b3a43fa172 > > system_test/replication_testsuite/testcase_0004/testcase_0004_properties.json > b91cef88533cb5e7b70a222fd924cd49b786bc93 > > system_test/replication_testsuite/testcase_0005/testcase_0005_properties.json > 4b3f76a21642dc568cc343f5ffb3364968859503 > > system_test/replication_testsuite/testcase_0006/testcase_0006_properties.json > b9b3485c46b6193575873e50143b712fcfc45bd0 > > system_test/replication_testsuite/testcase_0007/testcase_0007_properties.json > 5c4351fad98ef0a81e7dd93f397cb8cd3e3439f8 > > system_test/replication_testsuite/testcase_0008/testcase_0008_properties.json > 79cfed8bc63e429ce4875e68e85e2ca72fb2f7e9 > > system_test/replication_testsuite/testcase_0009/testcase_0009_properties.json > a52b709579f58c6c7766b07691c41a971bb4dd95 > > system_test/replication_testsuite/testcase_0010/testcase_0010_properties.json > 8d4b5fea2d1f91bced397af14d47b0e24ee04872 > > system_test/replication_testsuite/testcase_0011/testcase_0011_properties.json > b03f9cf059899ce42e0250f084b5a445ec4aa8bc > > system_test/replication_testsuite/testcase_0021/testcase_0021_properties.json > 40c2f8df13051ed90eb1ddb6a2005b43518213f5 > > system_test/replication_testsuite/testcase_0022/testcase_0022_properties.json > 7a3046035329f8986339fe25c0da4c2ecaa9b1ea > > system_test/replication_testsuite/testcase_0023/testcase_0023_properties.json > d921f01f332a4ed810b3ce12540a52892f89f178 > > system_test/replication_testsuite/testcase_0024/testcase_0024_properties.json > 839eb1476981ead1a82ecc4c00703eb1eec208ef > > system_test/replication_testsuite/testcase_0101/testcase_0101_properties.json > 85e4b61751b12dc4a9092097c1ae13a5d8947dee > > system_test/replication_testsuite/testcase_0102/testcase_0102_properties.json > 0d2f59fbcaa30159e26e01e83d9957039702a902 > > system_test/replication_testsuite/testcase_0103/testcase_0103_properties.json > 34acfa9d5c684e6221f54ee32ce89f47088bbd5a > > system_test/replication_testsuite/testcase_0104/testcase_0104_properties.json > 4145345fc433a294ae4094b698bf8d1f3776367f > > system_test/replication_testsuite/testcase_0105/testcase_0105_properties.json > 2eecc768e7e9711dd733819d7111090137435783 > > system_test/replication_testsuite/testcase_0106/testcase_0106_properties.json > 744174eab1c82984f92bd8f120e1df2a4892cef0 > > system_test/replication_testsuite/testcase_0107/testcase_0107_properties.json > e881b138c1fc042666e99187270e851f821e84ee > > system_test/replication_testsuite/testcase_0108/testcase_0108_properties.json > 7b48fdb441ad5931c1146d3797a9e5dc46d27b94 > > system_test/replication_testsuite/testcase_0109/testcase_0109_properties.json > a98ae0390bf556357b867a3d0c50022406f78a63 > > system_test/replication_testsuite/testcase_0110/testcase_0110_properties.json > f51abc156ae634b268213e9d97a96565484b5269 > > system_test/replication_testsuite/testcase_0111/testcase_0111_properties.json > fff0d689eba5f31248be1f972e5481ac8acc3438 > > system_test/replication_testsuite/testcase_0112/testcase_0112_properties.json > 636f02a2342d8e9d35c14b321e61f76c74b07339 > > system_test/replication_testsuite/testcase_0113/testcase_0113_properties.json > bdb885e43829e3075c875ec587dbc78d01e1da87 > > system_test/replication_testsuite/testcase_0114/testcase_0114_properties.json > ca51c5f5f183fccb8e3400a429533bb66547eeb7 > > system_test/replication_testsuite/testcase_0115/testcase_0115_properties.json > 459becf251d7cf18eb60c9da7473abc5515c5706 > > system_test/replication_testsuite/testcase_0116/testcase_0116_properties.json > c9471bc62ac6ac02022a53738dfb15fa65105c71 > > system_test/replication_testsuite/testcase_0117/testcase_0117_properties.json > 8159464b16c2c478f7c34e567e7dc83ab0d89545 > > system_test/replication_testsuite/testcase_0118/testcase_0118_properties.json > f73f0e43a3cf7176202232e768005c1279775e47 > > system_test/replication_testsuite/testcase_0119/testcase_0119_properties.json > 442a481230b151cb267991d4d6bdacee2d49344f > > system_test/replication_testsuite/testcase_0121/testcase_0121_properties.json > 606aad399dd3a69f80f86284c5cc3854fc3d237d > > system_test/replication_testsuite/testcase_0122/testcase_0122_properties.json > ea475361c8c5493c9bd07a0bc0ec32b711bc3adb > > system_test/replication_testsuite/testcase_0123/testcase_0123_properties.json > f9c92733cf4656e621260f58a4ee6b3f4422158e > > system_test/replication_testsuite/testcase_0124/testcase_0124_properties.json > 02f8506b20d345c3d61c8deb2a8fd1bdccc86f28 > > system_test/replication_testsuite/testcase_0125/testcase_0125_properties.json > 00482791aa9699f8d4f7d0b8edfdd01fd2fbef36 > > system_test/replication_testsuite/testcase_0126/testcase_0126_properties.json > 5119e61bbfca3ae4a7d028d1a39150d422d3c900 > > system_test/replication_testsuite/testcase_0127/testcase_0127_properties.json > 8b53fa7df5056e3d1b8ff76e5615059e5932d833 > > system_test/replication_testsuite/testcase_0128/testcase_0128_properties.json > e8edb9f4c70099c89172beb7672f23ab7cfcb2b6 > > system_test/replication_testsuite/testcase_0131/testcase_0131_properties.json > a140882287b9cf3453167b01255b949a705a9f6e > > system_test/replication_testsuite/testcase_0132/testcase_0132_properties.json > 48b30c7e014f0b5d504af1a760599525863e8895 > > system_test/replication_testsuite/testcase_0133/testcase_0133_properties.json > 8276aae0aa8226461e4595a17068e63442ddbcf5 > > system_test/replication_testsuite/testcase_0134/testcase_0134_properties.json > 73bb85995226230bcef47d890f72ec1436f24878 > > system_test/replication_testsuite/testcase_0151/testcase_0151_properties.json > eebba4db5d978267daf3ebe00615ac7711fb29d3 > > system_test/replication_testsuite/testcase_0152/testcase_0152_properties.json > debf544b3541a7297a42a4aae478a0c1d315f216 > > system_test/replication_testsuite/testcase_0153/testcase_0153_properties.json > 57b7d985236c83d4405f4f6a5a492022364c0b64 > > system_test/replication_testsuite/testcase_0154/testcase_0154_properties.json > c09fab719459d4692832f269898162438f3de69d > > system_test/replication_testsuite/testcase_0155/testcase_0155_properties.json > dd5ac52f86b023682b437c1d934373f827e05bcd > > system_test/replication_testsuite/testcase_0156/testcase_0156_properties.json > 8236ca58994b5e81f477c9ac6df837694e7967a9 > > system_test/replication_testsuite/testcase_0157/testcase_0157_properties.json > a28bf81e2b79e61b0dc7ca3fcce4e1dd592b9007 > > system_test/replication_testsuite/testcase_0158/testcase_0158_properties.json > 3d6edbdb16c5838b3f1d12a20c5d200d03668c63 > > system_test/replication_testsuite/testcase_0159/testcase_0159_properties.json > 030c9e8f0560f49291bf426b99a5a691caff210c > > system_test/replication_testsuite/testcase_0201/testcase_0201_properties.json > c6f8a23f358c5f7ab23ac48ce9b78062188b3f48 > > system_test/replication_testsuite/testcase_0202/testcase_0202_properties.json > 7b1a4c25c73fb274dfa8aba190c079330af8998f > > system_test/replication_testsuite/testcase_0203/testcase_0203_properties.json > 47276a85a038e278de75f1775da9ec9ce7912695 > > system_test/replication_testsuite/testcase_0204/testcase_0204_properties.json > 3742cfad8f25149accb062d4f4b7d23de484bcf3 > > system_test/replication_testsuite/testcase_0205/testcase_0205_properties.json > d41672d0b281810fb0b2355a77433287b6f9d4ca > > system_test/replication_testsuite/testcase_0206/testcase_0206_properties.json > a32d8882deb14fe26e8bc42ba500c7ca465f0dc3 > > system_test/replication_testsuite/testcase_0207/testcase_0207_properties.json > 573787099f92c50d629358a94ca2fb7e7ae4c264 > > system_test/replication_testsuite/testcase_0208/testcase_0208_properties.json > c3d1d2cb90ce21f4fe268dfabe731ea76308652a > > system_test/replication_testsuite/testcase_0209/testcase_0209_properties.json > 8d2cebafa0fb33c6fc729eacbb83d02ae45132a4 > > system_test/replication_testsuite/testcase_0251/testcase_0251_properties.json > 76ea6775657c916b543d36882b35fae3be757841 > > system_test/replication_testsuite/testcase_0252/testcase_0252_properties.json > 672b3ff883f2705d204981bf4a5140f333e345f5 > > system_test/replication_testsuite/testcase_0253/testcase_0253_properties.json > 19bd096b837353fae59a133feb413b5589ddb385 > > system_test/replication_testsuite/testcase_0254/testcase_0254_properties.json > 8d6add098da11324c065c856700c1296b827297a > > system_test/replication_testsuite/testcase_0255/testcase_0255_properties.json > 2a7e7776214d6071b492353c28023754ec815d6c > > system_test/replication_testsuite/testcase_0256/testcase_0256_properties.json > 91fbc9da75e2998ae75fbec140301c671ff92631 > > system_test/replication_testsuite/testcase_0257/testcase_0257_properties.json > 5594a9bc76d1ce50ddbf1668549dadca8eea8e84 > > system_test/replication_testsuite/testcase_0258/testcase_0258_properties.json > 6eabd3772dbe643ebca1495f174d2a228eccda02 > > system_test/replication_testsuite/testcase_0259/testcase_0259_properties.json > 3bbeeb10adfd8153b99f24fb55896b08de917974 > > system_test/replication_testsuite/testcase_0301/testcase_0301_properties.json > 496340c574ec35f0d5bd82e072692aee8221f05c > > system_test/replication_testsuite/testcase_0302/testcase_0302_properties.json > beabc86e2cdbac9cd8c5770f1e8924fbcc3dbab4 > > system_test/replication_testsuite/testcase_0303/testcase_0303_properties.json > 0e8f5a45895a32a780096aa2de8f619221989f7c > > system_test/replication_testsuite/testcase_0304/testcase_0304_properties.json > 98a385556c7819eb846182ae1df45b7d652caf86 > > system_test/replication_testsuite/testcase_0305/testcase_0305_properties.json > e09ac9c09b971e5ddf5b70181f6a2b75f3f22c02 > > system_test/replication_testsuite/testcase_0306/testcase_0306_properties.json > bcfe91b08c7e2424b2e8f012591caa47c1004387 > > system_test/replication_testsuite/testcase_0307/testcase_0307_properties.json > da07a9e4751b53269942cb984f67c2d0f1f46cbc > > system_test/replication_testsuite/testcase_0308/testcase_0308_properties.json > 0fecafc1a49c6aae3d171ab9d8f08944f7e8af06 > > system_test/replication_testsuite/testcase_0309/testcase_0309_properties.json > 90bd40442cc95ba049d4d11a91547de92b1ee9ff > system_test/replication_testsuite/testcase_1/testcase_1_properties.json > 7e1b3fbf2d22bcb33cc2f8dd52060f3f728eec0a > > system_test/replication_testsuite/testcase_4001/testcase_4001_properties.json > d2ffd952416e7822d2be4e171b5b5f951759408b > > system_test/replication_testsuite/testcase_4002/testcase_4002_properties.json > c86525db84ad0f926b96cfc93a0f8508d3e59981 > > system_test/replication_testsuite/testcase_4003/testcase_4003_properties.json > b77e4fdf400172a0b1bca4436223f9fd3928d705 > > system_test/replication_testsuite/testcase_4004/testcase_4004_properties.json > e753327795162b07b628b620e0a8b9e42d955062 > > system_test/replication_testsuite/testcase_4005/testcase_4005_properties.json > 5468401c6b41d280185d005d132c7a045ef50620 > > system_test/replication_testsuite/testcase_4006/testcase_4006_properties.json > e5ab0a0e8698133037cbf879873fc57a864211d4 > > system_test/replication_testsuite/testcase_4007/testcase_4007_properties.json > 7aa6e9a72ca74858e824abe251c666606fe10e4d > > system_test/replication_testsuite/testcase_4008/testcase_4008_properties.json > 08aa108f4f43a02424780cef4285383978ec8642 > > system_test/replication_testsuite/testcase_4011/testcase_4011_properties.json > 512fafb949c0d4b3a54030a5b87faeefca8c7ed2 > > system_test/replication_testsuite/testcase_4012/testcase_4012_properties.json > 9b711af013994357f2cc3a3046ba319326f9c1d8 > > system_test/replication_testsuite/testcase_4013/testcase_4013_properties.json > 3836366b3b19e5ab48c2e101c23978a165dbe4ba > > system_test/replication_testsuite/testcase_4014/testcase_4014_properties.json > 86ab75a46a48e1d99e42d8a9fc920b77d570da0b > > system_test/replication_testsuite/testcase_4015/testcase_4015_properties.json > 82d51b6072c6d6aaec827e357c82d4932ee60f96 > > system_test/replication_testsuite/testcase_4016/testcase_4016_properties.json > 31c1be0d73393c6e5fe72afff8494030f6d61eb9 > > system_test/replication_testsuite/testcase_4017/testcase_4017_properties.json > 72f78b074243ffabefe59c7158d3b45cda8593ea > > system_test/replication_testsuite/testcase_4018/testcase_4018_properties.json > ee459f4e35d682811a89bb8952d8d1afbff8e62d > > system_test/replication_testsuite/testcase_9051/testcase_9051_properties.json > 958eef7df0be16b5a3d6f126a5272b3dc3bffe1f > system_test/utils/kafka_system_test_utils.py > 5d2b7dfd46b07ab8749b1b2293e35183f14ae9af > > Diff: https://reviews.apache.org/r/18774/diff/ > > > Testing > ------- > > Rerun all system test. There is only one test in replication_suite that's > failing. Will file a separate jira. > > > Thanks, > > Jun Rao > >
