[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396834367 @sijie Ok thanks very much for the explanation, I already merged to my branch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396810024 @sijie I still have questions, 1. Although the fix will solve the object recycle problem, but the response with valid data is ignored. Will this case cause problem to the application using dl as a replication service? How the application using the dl client shoud do to handle this situation? Is there any retry mechanism embeded by the bookkeeper client? I think as a quick fix, if this happed the client should throw a exception to the application, so that the application can implement its own retry mechanism. Maybe we should not treat this situation ( the response with valid data come after response withoud data situation) as an error one.When we receive an empty reponse, we should not close the request? Or we should impletement a retry mechanism after this situation happend? 2. back to the fix detail, if request.complete is not called, then on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127 ``` writeSet.recycle(); orderedEnsemble.recycle(); ``` is not executed. will this cause resource leak? 3. if requestComplete already set the `buffer.retain();` [here](https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127) is redundant This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396810024 @sijie I still have questions, 1. Although the fix will solve the object recycle problem, but the response with valid data is ignored. Will this case cause problem to the application using dl as a replication service? How the application using the dl client shoud do to handle this situation? Is there any retry mechanism embeded by the bookkeeper client? I think as a quick fix, if this happed the client should throw a exception to the application, so that the application can implement its own retry mechanism. Maybe we should not treat this situation ( the response with valid data come after response withoud data situation) as an error one, when we receive an empty reponse, we should not close the request? Or we should impletement a retry mechanism after this situation happend? 2. back to the fix detail, if request.complete is not called, then on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127 ``` writeSet.recycle(); orderedEnsemble.recycle(); ``` is not executed. will this cause resource leak? 3. if requestComplete already set the `buffer.retain();` [here](https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127) is redundant This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie it's possible that compleRequest() come before request.complte(), such as https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will be closed in completeRequest() by the submitCallback() in it, the entryImpl will be recycled. Because completeRequest() does not set `comlete` to true, later when request.complete being called, ``` writeSet.recycle(); orderedEnsemble.recycle(); ``` will be executed,and although this time request.close will not be called. Since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie it's possible that compleRequest() come before request.complte(), such as thehttps://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will be closed in completeRequest() by the submitCallback() in it, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie it's possible that compleRequest() come before request.complte(), such as https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will be closed in completeRequest() by the submitCallback() in it, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie sometimes compleRequest() come before request.complte(), such as the path on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will be closed in completeRequest() by the submitCallback() in it, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie but some times compleRequest() come before request.complte(), such as the path on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will be closed in completeRequest() by the submitCallback() in it, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie but some times compleRequest() come before request.complte(), such as the path on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will first close in completeRequest() by the submitCallback() in it, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie but some times compleRequest() come before request.complte, such as the path on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will first close in requestComplete, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie but some times compleRequest() come before request.complte(), such as the path on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will first close in requestComplete, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie but some times requestComplete come before request.complte, such as the path on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L589, this will happen when the response without piggyback data comes before the response with data. In this case, the request will first close in requestComplete, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie but some times requestComplete come before request.complte, such as the path on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L584 , this will happen when the response without piggyback data comes before the response with data. In this case, the request will first close in requestComplete, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined even the operation is executed in the same thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396580056 @sijie but some times requestComplete come before request.complte, such as the path on https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L584 , this will happen when the response without piggyback data comes before the response with data. In this case, the request will first close in requestComplete, the entryImpl will be recycled. Then the request.complete will be called. although this time request.close will not be called, but since entryImpl is already recycled , if entryImpl is resued by other request, then data will be ruined. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396500590 @sijie but in https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L615 use `requestComplete` to guarantee request is only close once, while https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127 use `complete` to guarantee ,they are two diff variables. So it's possible after the request is closed at the same time entryImpl is recycled, the https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127 code will be executed, and the data maybe ruined. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396500590 @sijie but in https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L615 use `requestComplete` to guarantee request is only close once, while https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127 use `complete` to guarantee , it's two diff variables. So it's possible after the request is closed at the same time entryImpl is recycled, the https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127 code will be executed, and the data maybe ruined. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396500590 @sijie but in https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L617 use `requestComplete` to guarantee request is only close once, while https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127 use `complete` to guarantee , it's two diff variables. So it's possible after the request is closed at the same time entryImpl is recycled, the https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedAndEntryOp.java#L127 code will be executed, and the data maybe ruined. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396449927 @sijie I think the fix of infodog/bookkeeper#1 may have problem. because `request.complete(rCtx.getBookieIndex(), bookie, buffer, entryId) `will be called even the request is already closed. In request.complete ``` entryImpl.setLength(buffer.getLong(DigestManager.METADATA_LENGTH - 8)); entryImpl.setEntryBuf(content); ``` the entryImpl may already recycled and owned by another request, so this will ruin another request's data? Maybe should move ``` writeSet.recycle(); orderedEnsemble.recycle(); ``` from request.complete to request.close, and when we found thre requested is closed, we dont call request.complete? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396449927 @sijie I think the fix of infodog/bookkeeper#1 may have problem. because `request.complete(rCtx.getBookieIndex(), bookie, buffer, entryId) `will be called even the request is already closed. In request.complete ``` entryImpl.setLength(buffer.getLong(DigestManager.METADATA_LENGTH - 8)); entryImpl.setEntryBuf(content); ``` the entryImpl may already recycled and owned by another request, so this will ruin another request's data? Maybe should move ``` writeSet.recycle(); orderedEnsemble.recycle(); ``` from request.complete to request.close? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue. URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396449927 @sijie I think the fix of infodog/bookkeeper#1 may have problem. because `request.complete(rCtx.getBookieIndex(), bookie, buffer, entryId) `will be called even the request is already closed. In request.complete ``` entryImpl.setLength(buffer.getLong(DigestManager.METADATA_LENGTH - 8)); entryImpl.setEntryBuf(content); ``` the entryImpl may already recycled and owned by another request, so this will ruin another request's data? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue.
URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396167544
@eolivelli
I setup a bookkeeper cluster with 3 bookie server, and then setup two
clients read the logs , the client code is as following:
```
void downloadLog() throws IOException {
long nextTxId = getLocalLastTxId();
long lastShardTxId = 0;
try {
lastShardTxId = dlm.getLastTxId();
} catch (Throwable t) {
//当日志为空的时候，dlm.getLastTxId会出异常，所以需要try住
//do nothing
}
LogReader reader = dlm.getInputStream(nextTxId);
while (true) {
try {
while (true) {
try {
LogRecord record = reader.readNext(false);
if (record == null) {
break;
}
System.out.println("read record, txid=" +
record.getTransactionId() + ", lastShardId=" + lastShardTxId);
if (record.getTransactionId() > getLocalLastTxId()) {
updateLocalLog(record);
}
// read next record
} catch (LogEmptyException t) {
//没事
System.out.println("no record in log.");
} catch (Throwable ioe) {
// handle the exception
nextTxId = getLocalLastTxId();
reader = dlm.getInputStream(nextTxId);
}
}
if (isSwitchingToMaster) {
return;
}
synchronized (downloadThreadMonitor) {
downloadThreadMonitor.wait(100);
}
if (isSwitchingToMaster) {
return;
}
} catch (Throwable e) {
e.printStackTrace();
}
}
}
```
when I start a writer write to the bookkeeper server, after serveral
thoundsand writes, the client will throw the exception saying that object
recycled , then the client will not receive any new data. In fact I start 2
clients in the same time to read from the server,and always one client fails,
randomly.
After apply the fix, my client will run and always get the updated data. So
I think the fix addressed the problem.
Maybe we can create a test case like this? But I dont know how to do that.
By the way, When I debug the problem, I add some log statements to the code,
and the logs shows there is still some problem hiding. Still shows something I
can't understand. Maybe I am not look carefully enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue.
URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396167544
@eolivelli
I setup a bookkeeper cluster with 3 bookie server, and then setup a client
read the logs , the client code is as following:
```
void downloadLog() throws IOException {
long nextTxId = getLocalLastTxId();
long lastShardTxId = 0;
try {
lastShardTxId = dlm.getLastTxId();
} catch (Throwable t) {
//当日志为空的时候，dlm.getLastTxId会出异常，所以需要try住
//do nothing
}
LogReader reader = dlm.getInputStream(nextTxId);
while (true) {
try {
while (true) {
try {
LogRecord record = reader.readNext(false);
if (record == null) {
break;
}
System.out.println("read record, txid=" +
record.getTransactionId() + ", lastShardId=" + lastShardTxId);
if (record.getTransactionId() > getLocalLastTxId()) {
updateLocalLog(record);
}
// read next record
} catch (LogEmptyException t) {
//没事
System.out.println("no record in log.");
} catch (Throwable ioe) {
// handle the exception
nextTxId = getLocalLastTxId();
reader = dlm.getInputStream(nextTxId);
}
}
if (isSwitchingToMaster) {
return;
}
synchronized (downloadThreadMonitor) {
downloadThreadMonitor.wait(100);
}
if (isSwitchingToMaster) {
return;
}
} catch (Throwable e) {
e.printStackTrace();
}
}
}
```
when I start a writer write to the bookkeeper server, with serveral
thoundsand writes, the client will throw the exception, saying that the object
recycled problem, then the client will not receive any new data. In fact I
start 2 clients in the same time to read from the server. And always one client
fails, randomly.
After apply the fix, my client will run and always get the updated data. So
I think the fix addressed the problem.
Maybe we can create a test case like this? But I dont know how to do that.
By the way, When I debug the problem, I add some log statements to the code,
and the logs shows there is still some problem hiding. Still shows something I
can't understand. Maybe I am not look carefully enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue.
URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396167544
I setup a bookkeeper cluster with 3 bookie server, and then setup a client
read the logs , the client code is as following:
```
void downloadLog() throws IOException {
long nextTxId = getLocalLastTxId();
long lastShardTxId = 0;
try {
lastShardTxId = dlm.getLastTxId();
} catch (Throwable t) {
//当日志为空的时候，dlm.getLastTxId会出异常，所以需要try住
//do nothing
}
LogReader reader = dlm.getInputStream(nextTxId);
while (true) {
try {
while (true) {
try {
LogRecord record = reader.readNext(false);
if (record == null) {
break;
}
System.out.println("read record, txid=" +
record.getTransactionId() + ", lastShardId=" + lastShardTxId);
if (record.getTransactionId() > getLocalLastTxId()) {
updateLocalLog(record);
}
// read next record
} catch (LogEmptyException t) {
//没事
System.out.println("no record in log.");
} catch (Throwable ioe) {
// handle the exception
nextTxId = getLocalLastTxId();
reader = dlm.getInputStream(nextTxId);
}
}
if (isSwitchingToMaster) {
return;
}
synchronized (downloadThreadMonitor) {
downloadThreadMonitor.wait(100);
}
if (isSwitchingToMaster) {
return;
}
} catch (Throwable e) {
e.printStackTrace();
}
}
}
```
when I start a writer write to the bookkeeper server, with serveral
thoundsand writes, the client will throw the exception, saying that the object
recycled problem, then the client will not receive any new data. In fact I
start 2 clients in the same time to read from the server. And always one client
fails, randomly.
After apply the fix, my client will run and always get the updated data. So
I think the fix addressed the problem.
Maybe we can create a test case like this? But I dont know how to do that.
By the way, When I debug the problem, I add some log statements to the code,
and the logs shows there is still some problem hiding. Still shows something I
can't understand. Maybe I am not look carefully enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] infodog commented on issue #1509: solve #1476 issue.
infodog commented on issue #1509: solve #1476 issue.
URL: https://github.com/apache/bookkeeper/pull/1509#issuecomment-396167544
I setup a bookkeeper cluster with 3 bookie server, and then setup a client
read the logs , the client code is as following:
```
void downloadLog() throws IOException {
long nextTxId = getLocalLastTxId();
long lastShardTxId = 0;
try {
lastShardTxId = dlm.getLastTxId();
} catch (Throwable t) {
//当日志为空的时候，dlm.getLastTxId会出异常，所以需要try住
//do nothing
}
LogReader reader = dlm.getInputStream(nextTxId);
while (true) {
try {
while (true) {
try {
LogRecord record = reader.readNext(false);
if (record == null) {
break;
}
System.out.println("read record, txid=" +
record.getTransactionId() + ", lastShardId=" + lastShardTxId);
if (record.getTransactionId() > getLocalLastTxId()) {
updateLocalLog(record);
}
// read next record
} catch (LogEmptyException t) {
//没事
System.out.println("no record in log.");
} catch (Throwable ioe) {
// handle the exception
nextTxId = getLocalLastTxId();
reader = dlm.getInputStream(nextTxId);
}
}
if (isSwitchingToMaster) {
return;
}
synchronized (downloadThreadMonitor) {
downloadThreadMonitor.wait(100);
}
if (isSwitchingToMaster) {
return;
}
} catch (Throwable e) {
e.printStackTrace();
}
}
}
```
when I start a writer write to the bookkeeper server, with serveral
thoundsand writes, the client will throw the exception, saying that the object
recycled problem, then the client will not receive any new data. In fact I
start 2 clients in the same time to read from the server. And always one client
fails, randomly.
After apply the fix, my client will run and always get the updated data. So
I think the fix addressed the data.
Maybe we can create a test case like this? But I dont know how to do that.
By the way, When I debug the problem, I add some log statements to the code,
and the logs shows there is still some problem hiding. Still shows something I
can't understand. Maybe I am not look carefully enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
