acelyc111 commented on PR #1618:
URL:
https://github.com/apache/incubator-pegasus/pull/1618#issuecomment-1732248282
I've also tested this patch on real HDFS cluster by specifying the configs:
```
[hdfs_test]
test_name_node = x.x.x.x
test_backup_path = /path/to/backup
```
sample result:
```
[==========] Running 12 tests from 4 test cases.
[----------] Global test environment set-up.
[----------] 4 tests from block_service_manager_test
[ RUN ] block_service_manager_test.do_download_remote_file_not_exist
E2023-09-23 07:59:44.467 (1695455984467163927 81633)
replica.THREAD_POOL_DEFAULT0.00003ed200010001:
block_service_manager.cpp:180:download_file(): download
file(test_dir/test_file) failed, file on remote file provider is damaged
[ OK ] block_service_manager_test.do_download_remote_file_not_exist (3
ms)
[ RUN ] block_service_manager_test.do_download_same_name_file
[ OK ] block_service_manager_test.do_download_same_name_file (2 ms)
[ RUN ] block_service_manager_test.do_download_file_exist
[ OK ] block_service_manager_test.do_download_file_exist (0 ms)
[ RUN ] block_service_manager_test.do_download_succeed
[ OK ] block_service_manager_test.do_download_succeed (1 ms)
[----------] 4 tests from block_service_manager_test (7 ms total)
[----------] 2 tests from FDSClientTest
[ RUN ] FDSClientTest.test_basic_operation
[ OK ] FDSClientTest.test_basic_operation (15 ms)
[ RUN ] FDSClientTest.test_concurrent_upload_download
[ OK ] FDSClientTest.test_concurrent_upload_download (14 ms)
[----------] 2 tests from FDSClientTest (30 ms total)
[----------] 2 tests from local_service
[ RUN ] local_service.store_metadata
[ OK ] local_service.store_metadata (0 ms)
[ RUN ] local_service.load_metadata
W2023-09-23 07:59:44.502 (1695455984502974495 81633)
replica.THREAD_POOL_DEFAULT0.00003ed200010001:
local_service.cpp:62:file_metadata_from_json(): decode meta data from json
failed: [json.exception.parse_error.101] parse error at line 1, column 1:
syntax error while parsing value - invalid literal; last read: 'i' [invalid]
W2023-09-23 07:59:44.503 (1695455984503600402 81633)
replica.THREAD_POOL_DEFAULT0.00003ed200010001:
local_service.cpp:62:file_metadata_from_json(): decode meta data from json
failed: [json.exception.parse_error.101] parse error at line 1, column 19:
syntax error while parsing object key - invalid string: missing closing quote;
last read: '"no'; expected string literal [{"md5":"abcde","no]
[ OK ] local_service.load_metadata (2 ms)
[----------] 2 tests from local_service (2 ms total)
[----------] 4 tests from HDFSClientTest
[ RUN ] HDFSClientTest.test_hdfs_read_write/0
23/09/23 15:59:45 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
clean up all old files.
E2023-09-23 07:59:45.699 (1695455985699864853 81638)
replica.THREAD_POOL_BLOCK_SERVICE2.0101000000000006:
hdfs_service.cpp:248:operator()(): HDFS remove_path failed: path
/tmp/hdfs_client_test not found.
test write operation.
W2023-09-23 07:59:45.710 (1695455985710805522 81639)
replica.THREAD_POOL_BLOCK_SERVICE3.0101000000000008:
hdfs_service.cpp:286:get_file_meta(): HDFS file
/tmp/hdfs_client_test/test_write_file does not exist.
test read just written contents.
23/09/23 15:59:46 WARN hdfs.DFSClient: zero
23/09/23 15:59:46 WARN hdfs.DFSClient: zero
[ OK ] HDFSClientTest.test_hdfs_read_write/0 (1744 ms)
[ RUN ] HDFSClientTest.test_upload_and_download/0
clean up all old files.
create and upload: hdfs_client_test/test_file.
test download hdfs_client_test/test_file.
23/09/23 15:59:46 WARN hdfs.DFSClient: zero
[ OK ] HDFSClientTest.test_upload_and_download/0 (554 ms)
[ RUN ] HDFSClientTest.test_concurrent_upload_download/0
clean up all old files.
E2023-09-23 07:59:49.119 (1695455989119204768 81640)
replica.THREAD_POOL_BLOCK_SERVICE4.010100000000001b:
hdfs_service.cpp:248:operator()(): HDFS remove_path failed: path
/tmp/hdfs_concurrent_test not found.
test concurrent upload files.
file randomfile0 upload finished.
file randomfile2 upload finished.
file randomfile6 upload finished.
...
file randomfile63_d download finished
file randomfile61_d download finished
file randomfile62_d download finished
[ OK ] HDFSClientTest.test_concurrent_upload_download/0 (5198 ms)
[ RUN ] HDFSClientTest.test_rename_path_while_writing/0
[ OK ] HDFSClientTest.test_rename_path_while_writing/0 (62 ms)
[----------] 4 tests from HDFSClientTest (7559 ms total)
[----------] Global test environment tear-down
[==========] 12 tests from 4 test cases ran. (7598 ms total)
[ PASSED ] 12 tests.
dsn exit with code 0
~/dev/pegasus_review
Test finish time: Sat Sep 23 07:59:52 Asia 2023
Test elapsed time: 0m 12s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
