[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411708#comment-16411708
 ] 

ASF subversion and git services commented on CLOUDSTACK-10320:
--

Commit ca1760a46b67004d049b34ffbae4b635ec0a1561 in cloudstack's branch 
refs/heads/master from [~marcaurele]
[ https://gitbox.apache.org/repos/asf?p=cloudstack.git;h=ca1760a ]

CLOUDSTACK-10320 - Invalid pair for response object breaking response parsing 
(#2481)



> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411705#comment-16411705
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

DaanHoogland closed pull request #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java 
b/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
index 304a122a0b7..442d3cc181d 100644
--- a/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
+++ b/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
@@ -1315,6 +1315,10 @@ protected void addJoins(StringBuilder str, 
Collection searchAndCount(final SearchCriteria sc, 
final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getCount(sc);
+// Count cannot be less than the result set but can be higher due to 
pagination, see CLOUDSTACK-10320
+if (count < objects.size()) {
+count = objects.size();
+}
 return new Pair(objects, count);
 }
 
@@ -1323,6 +1327,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
+// Cannot assume if it's more than one since the count is distinct 
vs search
+count = 1;
+}
 return new Pair(objects, count);
 }
 
@@ -1331,6 +1340,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter, final String[] distinctColumns) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc, distinctColumns);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
+// Cannot assume if it's more than one since the count is distinct 
vs search
+count = 1;
+}
 return new Pair(objects, count);
 }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> 

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409843#comment-16409843
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-375373737
 
 
   @DaanHoogland bad news: it didn't last very long before I had to rollback 
the isolation level due to DB dead locks (at least in our branch based from 
4.4.2). So it's the only quick fix & hacky solution for now which doesn't add 
another issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406243#comment-16406243
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-374577255
 
 
   I don't think we're in a hurry to fix the problem. I will investigate on our 
platform and maybe try to narrow down the isolation level to those queries 
only. It will take one or two weeks to get the result. Anyway, I don't see 
another option to fix the problem with the same requirements (total count, 
pagination).
   The other solution would be to get rid of the count value in the API 
response, making this change obsolete. It should be a point to keep in mind for 
the next API version.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405923#comment-16405923
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

DaanHoogland commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-374505072
 
 
   Good experiment @marcaurele , I have no stress test env at my disposal so i 
can not verify. I am worried that my question is blocking your fix. Let's try 
to separate the fix and the improvement.
   Is the transaction failing as a solution to the bug if the isolation level 
isn't changed as well?
   * If it is, let's merge as is.
   * If it isn't (but performing badly) let's merge it as transaction.
   In both cases we can create a ticket to track it further, ok?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405057#comment-16405057
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-374269828
 
 
   The default isolation level in CS is `read-committed` 
(https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/TransactionLegacy.java#L1044)
 but not `repeatable-read`, except if you explicitly set it in the 
`db.properties` file. If the default value is change, and I modify the PR to 
use a transaction around those 2 SQL read operations, the issue is fixed, but...
   
   Changing the default isolation level will certainly have an impact on 
performance on the database 
(https://www.percona.com/blog/2015/01/14/mysql-performance-implications-of-innodb-isolation-modes/).
 So, I'll change the PR to only have a repeatable-read isolation for those set 
of methods, and will try to measure how it performs.
   
   If people are willing to test in their environment (quite loaded) the global 
isolation level change, it would be interesting to hear the performance 
outcomes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404551#comment-16404551
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on a change in pull request #2481: CLOUDSTACK-10320 - 
Invalid pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r175355297
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1323,6 +1327,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
+// Cannot assume if it's more than one since the count is distinct 
vs search
+count = 1;
 
 Review comment:
   Because the search is mostly done against views which don't return distinct 
rows due to the *not so smart* queries fetching the ~details~ resource tags 
rows. Look at the APIHelper with iterate on duplicate row entries, but only 
append the details.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404488#comment-16404488
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-374133561
 
 
   @DaanHoogland you're right and I looked again on how to achieve this 
isolation between queries, with a `repeatable-read` but so far I didn't manage 
to have it correctly working. I read that for innodb, the default isolation in 
a transaction is the `repeatable-read` but my local debugging tests did not 
work as expected. I will dig more to find a better solution and not a hacky one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404484#comment-16404484
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on a change in pull request #2481: CLOUDSTACK-10320 - 
Invalid pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r175355994
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1315,6 +1315,10 @@ protected void addJoins(StringBuilder str, 
Collection searchAndCount(final SearchCriteria sc, 
final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getCount(sc);
+// Count cannot be less than the result set but can be higher due to 
pagination, see CLOUDSTACK-10320
+if (count < objects.size()) {
 
 Review comment:
   I will change how it's handled


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404481#comment-16404481
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on a change in pull request #2481: CLOUDSTACK-10320 - 
Invalid pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r17538
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1331,6 +1340,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter, final String[] distinctColumns) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc, distinctColumns);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
+// Cannot assume if it's more than one since the count is distinct 
vs search
 
 Review comment:
   See comment before


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404480#comment-16404480
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on a change in pull request #2481: CLOUDSTACK-10320 - 
Invalid pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r175355499
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1323,6 +1327,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
+// Cannot assume if it's more than one since the count is distinct 
vs search
+count = 1;
 
 Review comment:
   Moreover, the search might return a page size list, while the count should 
be global: listVirtualMachines returns a count of 234 and you have 50 entries 
in the search based on the `page` and `pageSize`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404475#comment-16404475
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on a change in pull request #2481: CLOUDSTACK-10320 - 
Invalid pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r175355297
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1323,6 +1327,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
+// Cannot assume if it's more than one since the count is distinct 
vs search
+count = 1;
 
 Review comment:
   Because the search is mostly done against views which don't return distinct 
rows due to the *not so smart* queries fetching the details rows. Look at the 
APIHelper with iterate on duplicate row entries, but only append the details.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404421#comment-16404421
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

rhtyd commented on a change in pull request #2481: CLOUDSTACK-10320 - Invalid 
pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r175338869
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1315,6 +1315,10 @@ protected void addJoins(StringBuilder str, 
Collection searchAndCount(final SearchCriteria sc, 
final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getCount(sc);
+// Count cannot be less than the result set but can be higher due to 
pagination, see CLOUDSTACK-10320
+if (count < objects.size()) {
 
 Review comment:
   This logic may be used in changes below.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404419#comment-16404419
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

rhtyd commented on a change in pull request #2481: CLOUDSTACK-10320 - Invalid 
pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r175338811
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1323,6 +1327,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
+// Cannot assume if it's more than one since the count is distinct 
vs search
+count = 1;
 
 Review comment:
   why not put count = objects.size() ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404420#comment-16404420
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

rhtyd commented on a change in pull request #2481: CLOUDSTACK-10320 - Invalid 
pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r175338839
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1331,6 +1340,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter, final String[] distinctColumns) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc, distinctColumns);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
+// Cannot assume if it's more than one since the count is distinct 
vs search
 
 Review comment:
   Why not put count = objects.size() ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404401#comment-16404401
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

blueorangutan commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-374107511
 
 
   Trillian test result (tid-2388)
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 33025 seconds
   Marvin logs: 
https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2481-t2388-kvm-centos7.zip
   Intermitten failure detected: /marvin/tests/smoke/test_certauthority_root.py
   Intermitten failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermitten failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermitten failure detected: /marvin/tests/smoke/test_templates.py
   Intermitten failure detected: /marvin/tests/smoke/test_usage.py
   Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
   Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermitten failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
   Smoke tests completed. 62 look OK, 5 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_04_extract_template | `Failure` | 128.45 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   test_06_download_detached_volume | `Failure` | 138.31 | test_volumes.py
   test_04_rvpc_network_garbage_collector_nics | `Failure` | 484.70 | 
test_vpc_redundant.py
   test_hostha_enable_ha_when_host_in_maintenance | `Error` | 2.86 | 
test_hostha_kvm.py
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404156#comment-16404156
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

blueorangutan commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-374039824
 
 
   @DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has 
been kicked to run smoke tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404154#comment-16404154
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

DaanHoogland commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-374039766
 
 
   @blueorangutan test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402078#comment-16402078
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

blueorangutan commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-373756211
 
 
   Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1787


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402011#comment-16402011
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

blueorangutan commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-373740886
 
 
   @rafaelweingartner a Jenkins job has been kicked to build packages. I'll 
keep you posted as I make progress.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402008#comment-16402008
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

rafaelweingartner commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-373740594
 
 
   @blueorangutan package


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397240#comment-16397240
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

DaanHoogland commented on issue #2481: CLOUDSTACK-10320 - Invalid pair for 
response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#issuecomment-372735409
 
 
   nice find, dirty hack, though maybe justified. Isn't the proper solution to 
put both queries in a transaction, @marcaurele ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395089#comment-16395089
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele commented on a change in pull request #2481: CLOUDSTACK-10320 - 
Invalid pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r173764470
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1323,6 +1327,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
 
 Review comment:
   `search` always return a list, never `null`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395053#comment-16395053
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

rafaelweingartner commented on a change in pull request #2481: CLOUDSTACK-10320 
- Invalid pair for response object breaking response parsing
URL: https://github.com/apache/cloudstack/pull/2481#discussion_r173751610
 
 

 ##
 File path: framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
 ##
 @@ -1323,6 +1327,11 @@ protected void addJoins(StringBuilder str, 
Collection searchAndDistinctCount(final 
SearchCriteria sc, final Filter filter) {
 List objects = search(sc, filter, null, false);
 Integer count = getDistinctCount(sc);
+// Count cannot be 0 if there is at least a result in the list, see 
CLOUDSTACK-10320
+if (count == 0 && !objects.isEmpty()) {
 
 Review comment:
   Is it possible for that `objects` variable there to assume a null value? If 
not, everything seems to be in order here. Otherwise, it might be better to use 
CollectionUtils.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLOUDSTACK-10320) Invalid pair for response object breaking response parsing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391422#comment-16391422
 ] 

ASF GitHub Bot commented on CLOUDSTACK-10320:
-

marcaurele opened a new pull request #2481: CLOUDSTACK-10320 - DAO: count value 
might be invalid regarding the result set returned
URL: https://github.com/apache/cloudstack/pull/2481
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Invalid pair for response object breaking response parsing
> --
>
> Key: CLOUDSTACK-10320
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10320
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: API
>Reporter: Marc-Aurèle Brothier
>Assignee: Marc-Aurèle Brothier
>Priority: Major
>
> Under some circumstances, the API is returning an invalid response, for 
> simplicity I will expose the JSON case. The API response on a 
> listVirtualMachines can be this string:
> {code:java}
> { "listvirtualmachinesresponse" :  ] } }{code}
> To understand how this is possible, assume you have more than one management 
> server and one is processing the destroy of a virtual machine in the account 
> X which is the only one it has. Another process is returning the result of 
> listVirtualMachines for that same account X. During the listVM command, the 
> result set is fetch with a searchAndDistinctCount due to the view 
> ([https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L1024).]
>  This is done through 2 queries in the GenericDao 
> [https://github.com/apache/cloudstack/blob/master/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1323]
>  and if you encounter the _right_ conditions, the VM will be marked as 
> removed in between those 2 queries. This results in having a Pair result with 
> at least one object but a count of 0. Then following how is done the 
> serialization of the response at 
> [https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/response/ApiResponseSerializer.java#L86]
>  you will reach the case where your output is the one previously mentioned.
> To overcome this issue, there isn't a true fix but only a better pair 
> response to ensure a correct response formatting. If the result set contains 
> at least something, the count cannot be 0 but we cannot guess the correct 
> answer, but only state it has at least one element.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)