[
https://issues.apache.org/jira/browse/AVRO-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kengo Seki updated AVRO-2562:
-----------------------------
Status: Patch Available (was: Open)
> Expose the JavaScript BlockEncoder's piped stream
> -------------------------------------------------
>
> Key: AVRO-2562
> URL: https://issues.apache.org/jira/browse/AVRO-2562
> Project: Apache Avro
> Issue Type: New Feature
> Components: javascript
> Reporter: Kengo Seki
> Assignee: Kengo Seki
> Priority: Major
>
> The "createFileEncoder" test for the JavaScript implementation is flaky. It
> sometimes fails in docker mode, as follows:
> {code}
> ~/repos/avro$ git checkout master
> ~/repos/avro$ ./build.sh docker
> sekikn@13d38772b8e6:~/avro$ ./build.sh veryclean
> sekikn@13d38772b8e6:~/avro$ cd lang/js
> sekikn@13d38772b8e6:~/avro/lang/js$ node -v
> v6.17.1
> sekikn@13d38772b8e6:~/avro/lang/js$ npm i
> sekikn@13d38772b8e6:~/avro/lang/js$ for i in $(seq 1 100); do echo "#${i}";
> if ! npm t; then break; fi; done
> #1
> > [email protected] test /home/sekikn/avro/lang/js
> > mocha
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> 381 passing (557ms)
> (snip)
> #7
> > [email protected] test /home/sekikn/avro/lang/js
> > mocha
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> 380 passing (554ms)
> 1 failing
> 1) files createFileEncoder:
> Uncaught AssertionError: 0 == 2
> + expected - actual
> -0
> +2
>
> at BlockDecoder.<anonymous> (test/test_files.js:574:18)
> at endReadableNT (_stream_readable.js:978:12)
> at _combinedTickCallback (internal/process/next_tick.js:80:11)
> at process._tickCallback (internal/process/next_tick.js:104:9)
> npm ERR! Test failed. See above for more details.
> {code}
> And it always fails with Node.js v8.10.0, which is installed by apt-get on
> Ubuntu 18.04 LTS.
> {code}
> ~/repos/avro$ git checkout master
> ~/repos/avro$ cd lang/js
> ~/repos/avro/lang/js$ ./build.sh test
> > [email protected] cover /home/sekikn/repos/avro/lang/js
> > istanbul cover _mocha
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
>
> ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
> 380 passing (537ms)
> 1 failing
> 1) files createFileEncoder:
> Uncaught AssertionError [ERR_ASSERTION]: 0 == 2
> + expected - actual
> -0
> +2
>
> at BlockDecoder.<anonymous> (test/test_files.js:574:18)
> at endReadableNT (_stream_readable.js:1064:12)
> at _combinedTickCallback (internal/process/next_tick.js:138:11)
> at process._tickCallback (internal/process/next_tick.js:180:9)
> =============================================================================
> Writing coverage object
> [/home/sekikn/repos/avro/lang/js/coverage/coverage.json]
> Writing coverage reports at [/home/sekikn/repos/avro/lang/js/coverage]
> =============================================================================
> =============================== Coverage summary
> ===============================
> Statements : 100% ( 2529/2529 )
> Branches : 100% ( 996/996 )
> Functions : 100% ( 414/414 )
> Lines : 100% ( 2443/2443 )
> ================================================================================
> npm ERR! Linux 5.0.0-25-generic
> npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "cover"
> npm ERR! node v8.10.0
> npm ERR! npm v3.5.2
> npm ERR! code ELIFECYCLE
> npm ERR! [email protected] cover: `istanbul cover _mocha`
> npm ERR! Exit status 1
> npm ERR!
> npm ERR! Failed at the [email protected] cover script 'istanbul cover _mocha'.
> npm ERR! Make sure you have the latest version of node.js and npm installed.
> npm ERR! If you do, this is most likely a problem with the avro-js package,
> npm ERR! not with npm itself.
> npm ERR! Tell the author that this fails on your system:
> npm ERR! istanbul cover _mocha
> npm ERR! You can get information on how to open an issue for this project
> with:
> npm ERR! npm bugs avro-js
> npm ERR! Or if that isn't available, you can get their info via:
> npm ERR! npm owner ls avro-js
> npm ERR! There is likely additional logging output above.
> npm ERR! Please include the following file with any support request:
> npm ERR! /home/sekikn/repos/avro/lang/js/npm-debug.log
> {code}
> In this test, {{createFileDecoder()}} is called in the event listener for the
> {{'finish'}} event emitted from the {{encoder}} object.
> {code:title=test/test_files.js}
> 553 test('createFileEncoder', function (cb) {
> 554 var type = createType({
> 555 type: 'record',
> 556 name: 'Person',
> 557 fields: [
> 558 {name: 'name', type: 'string'},
> 559 {name: 'age', type: 'int'}
> 560 ]
> 561 });
> 562 var path = tmp.fileSync().name;
> 563 var encoder = files.createFileEncoder(path, type);
> 564 encoder.write({name: 'Ann', age: 32});
> 565 encoder.end({name: 'Bob', age: 33});
> 566 var n = 0;
> 567 encoder.on('finish', function () {
> 568 files.createFileDecoder(path)
> 569 .on('data', function (obj) {
> 570 n++;
> 571 assert(type.isValid(obj));
> 572 })
> 573 .on('end', function () {
> 574 assert.equal(n, 2);
> 575 cb();
> 576 });
> 577 });
> 578 });
> {code}
> And the {{encoder}} object has the underlying writable stream created by
> {{createWriteStream()}}.
> {code:title=lib/files.js}
> 562 function createFileEncoder(path, schema, opts) {
> 563 var encoder = new BlockEncoder(schema, opts);
> 564 encoder.pipe(fs.createWriteStream(path, {defaultEncoding: 'binary'}));
> 565 return encoder;
> 566 }
> {code}
> In such a situation, the caller should wait for the underlying stream's
> {{'finish'}} event rather than {{encoder}}'s. Because when the {{encoder}}
> has finished to write and the caller catch its {{'finish'}} event, the
> underlying stream perhaps doesn't finished to write into the disk yet (cf.
> https://stackoverflow.com/questions/40864496/detecting-the-end-of-a-piped-stream-operation).
> I think BlockEncoder should expose a reference to the underlying piped
> stream, so that users can listen its event (typically {{'finish'}}).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
