[jira] [Commented] (SENTRY-1400) [Flaky test] TestSentryWebServerWithSSL times out
[
https://issues.apache.org/jira/browse/SENTRY-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987463#comment-15987463
]
Hadoop QA commented on SENTRY-1400:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12865409/SENTRY-1400.001-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2588/console
This message is automatically generated.
> [Flaky test] TestSentryWebServerWithSSL times out
> -
>
> Key: SENTRY-1400
> URL: https://issues.apache.org/jira/browse/SENTRY-1400
> Project: Sentry
> Issue Type: Test
>Affects Versions: sentry-ha-redesign
>Reporter: Sravya Tirukkovalur
>Assignee: Na Li
> Labels: test
> Attachments: SENTRY-1400.001-sentry-ha-redesign.patch,
> SENTRY-1400.001-sentry-ha-redesign.patch, Sentry-1649 Patch30 Test fail build
> 2569 a.zip
>
>
> I atleast saw two precommit jobs which failed due to this:
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console
> [~lili] any idea whats going on?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1400) [Flaky test] TestSentryWebServerWithSSL times out
[
https://issues.apache.org/jira/browse/SENTRY-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986043#comment-15986043
]
Hadoop QA commented on SENTRY-1400:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12865277/SENTRY-1400.001-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 3 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.persistent.TestSentryStore
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.persistent.TestSentryStore
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2581/console
This message is automatically generated.
> [Flaky test] TestSentryWebServerWithSSL times out
> -
>
> Key: SENTRY-1400
> URL: https://issues.apache.org/jira/browse/SENTRY-1400
> Project: Sentry
> Issue Type: Test
>Affects Versions: sentry-ha-redesign
>Reporter: Sravya Tirukkovalur
>Assignee: Na Li
> Labels: test
> Attachments: SENTRY-1400.001-sentry-ha-redesign.patch, Sentry-1649
> Patch30 Test fail build 2569 a.zip
>
>
> I atleast saw two precommit jobs which failed due to this:
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console
> [~lili] any idea whats going on?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1400) [Flaky test] TestSentryWebServerWithSSL times out
[ https://issues.apache.org/jira/browse/SENTRY-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985577#comment-15985577 ] Na Li commented on SENTRY-1400: --- In the console output, I can see most paths are "/home/jenkins/jenkins-slave/workspace/PreCommit-SENTRY-Build@2", but plugin is under "/home/jenkins/jenkins-slave/workspace/PreCommit-SENTRY-Build%402" Examples: "2017-04-26 00:54:34,026 (main) [DEBUG - net.sf.ehcache.config.DiskStoreConfiguration.translatePath(DiskStoreConfiguration.java:118)] Disk Store Path: /home/jenkins/jenkins-slave/workspace/PreCommit-SENTRY-Build@2/sentry-provider/sentry-provider-db/target/1493168072479/cache" "Debug is true storeKey true useTicketCache true useKeyTab true doNotPrompt true ticketCache is null isInitiator false KeyTab is /home/jenkins/jenkins-slave/workspace/PreCommit-SENTRY-Build@2/sentry-provider/sentry-provider-db/target/server.keytab refreshKrb5Config is true principal is sentry/localh...@example.com tryFirstPass is false useFirstPass is false storePass is false clearPass is false" "2017-04-26 00:54:33,974 (main) [DEBUG - net.sf.ehcache.config.ConfigurationFactory.parseConfiguration(ConfigurationFactory.java:96)] Configuring ehcache from URL: jar:file:/home/jenkins/jenkins-slave/workspace/PreCommit-SENTRY-Build%402/maven-repo/org/apache/directory/server/apacheds-core-api/2.0.0-M15/apacheds-core-api-2.0.0-M15.jar!/directory-cacheservice.xml" But > [Flaky test] TestSentryWebServerWithSSL times out > - > > Key: SENTRY-1400 > URL: https://issues.apache.org/jira/browse/SENTRY-1400 > Project: Sentry > Issue Type: Test >Reporter: Sravya Tirukkovalur >Assignee: Na Li > > I atleast saw two precommit jobs which failed due to this: > https://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console > https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console > [~lili] any idea whats going on? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1400) [Flaky test] TestSentryWebServerWithSSL times out
[
https://issues.apache.org/jira/browse/SENTRY-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985528#comment-15985528
]
Na Li commented on SENTRY-1400:
---
I suspect the cause of the issue is that
1) When the keystore path is set in SentryServiceIntegrationBase.setupConf()
line 186, the path is relative path.
"conf.set(ServerConfig.SENTRY_WEB_SSL_KEYSTORE_PATH,
Resources.getResource("keystore.jks").getPath());"
2) When SentryWebServer.start() is called, the key store path is used to get
keystore.jks, the absolute path of the key store is wrong because the root
directory is not the root where the keystore.jks is. In other words,
"/home/jenkins/jenkins-slave/workspace/PreCommit-SENTRY-Build%402" is not where
the directory in step 1) is.
the build cannot find
"/home/jenkins/jenkins-slave/workspace/PreCommit-SENTRY-Build%402/sentry-provider/sentry-provider-db/target/test-classes/keystore.jks
(No such file or directory)"
3) Some discussion online shows the slave node may use the wrong root
(top-level) directory.
https://issues.jenkins-ci.org/browse/JENKINS-17563
java.io.FileNotFoundException: Engineering/iOS/UScapeIt/UScapeIt.app.ipa (No
such file or directory)
"I believe the directory is fine.
The file reference should be relative to the workspace directory.
The job is running on a mac slave node.
Apparently, unlike the directory organization on the master node, it uses
workspace instead of jobs as a top-level directory.
Also, instead of having a workspace subdirectory, the directory matching the
jobname is the workspace."
> [Flaky test] TestSentryWebServerWithSSL times out
> -
>
> Key: SENTRY-1400
> URL: https://issues.apache.org/jira/browse/SENTRY-1400
> Project: Sentry
> Issue Type: Test
>Reporter: Sravya Tirukkovalur
>Assignee: Na Li
>
> I atleast saw two precommit jobs which failed due to this:
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console
> [~lili] any idea whats going on?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SENTRY-1400) [Flaky test] TestSentryWebServerWithSSL times out
[
https://issues.apache.org/jira/browse/SENTRY-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374216#comment-15374216
]
Li Li commented on SENTRY-1400:
---
Good catch!
It is weird. The jks file should be in
.../sentry-provider/sentry-provider-db/target/test-classes. Is it possible that
the keystore.jks file was deleted by another job?
Also you can verify the existence of jks file and TestSentryWebServerWithSSL in
your dev:
{code}
mvn clean install -DskipTests
cd sentry-provider/sentry-provider-db
mvn test
-Dtest=org.apache.sentry.provider.db.service.thrift.TestSentryWebServerWithSSL
{code}
> [Flaky test] TestSentryWebServerWithSSL times out
> -
>
> Key: SENTRY-1400
> URL: https://issues.apache.org/jira/browse/SENTRY-1400
> Project: Sentry
> Issue Type: Test
>Reporter: Sravya Tirukkovalur
>
> I atleast saw two precommit jobs which failed due to this:
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console
> [~lili] any idea whats going on?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1400) [Flaky test] TestSentryWebServerWithSSL times out
[
https://issues.apache.org/jira/browse/SENTRY-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374151#comment-15374151
]
Sravya Tirukkovalur commented on SENTRY-1400:
-
Thanks for taking a look [~lili]! Yeah, seems like other tests were rerun and
the reruns have passed, and thats the reason jenkins only reports the SSL test
which errored. For example:
{noformat}
Tests in error:
TestSentryWebServerWithSSL.setup:38->SentryServiceIntegrationBase.setup:103->SentryServiceIntegrationBase.startSentryService:117
ยป Timeout
Flaked tests:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceClientPool.testConnectionWithMultipleRetries(org.apache.sentry.provider.db.service.thrift.TestSentryServiceClientPool)
Run 1:
TestSentryServiceClientPool.testConnectionWithMultipleRetries:67->SentryServiceIntegrationBase.runTestAsSubject:347
Some tasks are failed.
Run 2: PASS
{noformat}
It seems like service might take more than a minute to startup, would it make
sense to bump this time out limit?
> [Flaky test] TestSentryWebServerWithSSL times out
> -
>
> Key: SENTRY-1400
> URL: https://issues.apache.org/jira/browse/SENTRY-1400
> Project: Sentry
> Issue Type: Test
>Reporter: Sravya Tirukkovalur
>
> I atleast saw two precommit jobs which failed due to this:
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console
> [~lili] any idea whats going on?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1400) [Flaky test] TestSentryWebServerWithSSL times out
[
https://issues.apache.org/jira/browse/SENTRY-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373681#comment-15373681
]
Li Li commented on SENTRY-1400:
---
The TestSentryWebServerWithSSL times out happened during "setup" method, which
is before running any test.
Also TestSentryWebServerWithSSL is not the only failure test in both precommit
jobs.
For example, before the failure of TestSentryWebServerWithSSL,
testConnectionWithMultipleRetries failed in
[precommit-sentry-build/1872|ttps://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console].
{code}
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 50.514 sec <<<
FAILURE! - in
org.apache.sentry.provider.db.service.thrift.TestSentryServiceClientPool
testConnectionWithMultipleRetries(org.apache.sentry.provider.db.service.thrift.TestSentryServiceClientPool)
Time elapsed: 0.889 sec <<< FAILURE!
java.lang.AssertionError: Some tasks are failed.
at
org.apache.sentry.provider.db.service.thrift.TestSentryServiceClientPool.testConnectionWithMultipleRetries(TestSentryServiceClientPool.java:67)
Running
org.apache.sentry.provider.db.service.thrift.TestAuthorizingDDLAuditLogWithKerberos
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.472 sec - in
org.apache.sentry.provider.db.service.thrift.TestAuthorizingDDLAuditLogWithKerberos
Running
org.apache.sentry.provider.db.service.thrift.TestSentryServiceWithInvalidMsgSize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.086 sec - in
org.apache.sentry.provider.db.service.thrift.TestSentryServiceWithInvalidMsgSize
Running
org.apache.sentry.provider.db.service.thrift.TestConnectionWithTicketTimeout
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.035 sec - in
org.apache.sentry.provider.db.service.thrift.TestConnectionWithTicketTimeout
Running org.apache.sentry.provider.db.service.thrift.TestSentryWebServerWithSSL
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 65.585 sec <<<
FAILURE! - in
org.apache.sentry.provider.db.service.thrift.TestSentryWebServerWithSSL
org.apache.sentry.provider.db.service.thrift.TestSentryWebServerWithSSL Time
elapsed: 65.584 sec <<< ERROR!
java.util.concurrent.TimeoutException: Server did not start after 60 seconds
at
org.apache.sentry.provider.db.service.thrift.TestSentryWebServerWithSSL.setup(TestSentryWebServerWithSSL.java:38)
{code}
And before the failure of TestSentryWebServerWithSSL, testRacingClients failed
in
[precommit-sentry-build/1874|https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console].
{code}
Running org.apache.sentry.service.thrift.TestLeaderStatus
Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 121.833 sec <<<
FAILURE! - in org.apache.sentry.service.thrift.TestLeaderStatus
testRacingClients(org.apache.sentry.service.thrift.TestLeaderStatus) Time
elapsed: 60.009 sec <<< ERROR!
java.lang.Exception: test timed out after 6 milliseconds
at
org.apache.sentry.service.thrift.TestLeaderStatus.testRacingClients(TestLeaderStatus.java:212)
testRacingClients(org.apache.sentry.service.thrift.TestLeaderStatus) Time
elapsed: 60.003 sec <<< ERROR!
java.lang.Exception: test timed out after 6 milliseconds
at
org.apache.sentry.service.thrift.TestLeaderStatus.testRacingClients(TestLeaderStatus.java:204)
{code}
So I think the root reason may not be related to TestSentryWebServerWithSSL.
> [Flaky test] TestSentryWebServerWithSSL times out
> -
>
> Key: SENTRY-1400
> URL: https://issues.apache.org/jira/browse/SENTRY-1400
> Project: Sentry
> Issue Type: Test
>Reporter: Sravya Tirukkovalur
>
> I atleast saw two precommit jobs which failed due to this:
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1872/console
> https://builds.apache.org/job/PreCommit-SENTRY-Build/1874/console
> [~lili] any idea whats going on?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
