[ 
https://issues.apache.org/jira/browse/AURORA-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Erb updated AURORA-1779:
--------------------------------
    Description: 
Steps to reproduce (if you are lucky):
{code}
vagrant destroy
vagrant up
vagrant ssh
cd /vagrant
./build-support/release/verify-release-candidate 0.16.0-rc1
{code}

For me, this resulted in the following initial stack trace (and a couple 
different follow ups due to a corrupted DB state):

{code}
I0921 19:45:38.730 [TaskEventBatchWorker, JobUpdateControllerImpl:359] 
Forwarding task change for vagrant/test/http_example/0 
I0921 19:45:38.744 [qtp1742582311-39, Slf4jRequestLog:60] 127.0.0.1 - - 
[21/Sep/2016:19:45:38 +0000] "POST //aurora.local/api HTTP/1.1" 200 108  
I0921 19:45:38.771 [TaskEventBatchWorker, JobUpdateControllerImpl:587] 
IJobUpdateKey{job=IJobKey{role=vagrant, environment=test, name=http_example}, 
id=97ff31cc-c17f-45ef-a1d9-c16cb07388c7} evaluation result: 
EvaluationResult{status=WORKING, 
sideEffects={0=SideEffect{action=Optional.of(ADD_TASK), statusChanges=[]}}} 
I0921 19:45:38.786 [TaskEventBatchWorker, JobUpdateControllerImpl:654] 
Executing side-effects for update of IJobUpdateKey{job=IJobKey{role=vagrant, 
environment=test, name=http_example}, id=97ff31cc-c17f-45ef-a1d9-c16cb07388c7}: 
{0=SideEffect{action=Optional.of(ADD_TASK), statusChanges=[]}} 
I0921 19:45:38.789 [TaskEventBatchWorker, InstanceActionHandler$AddTask:95] 
Adding instance IInstanceKey{jobKey=IJobKey{role=vagrant, environment=test, 
name=http_example}, instanceId=0} while ROLLING_FORWARD 
I0921 19:45:38.805 [TaskEventBatchWorker, StateMachine$Builder:389] 
vagrant-test-http_example-0-100b76ee-ed70-4e15-8e52-046f992d0b4a state machine 
transition INIT -> PENDING 
I0921 19:45:38.806 [TaskEventBatchWorker, TaskStateMachine:474] Adding work 
command SAVE_STATE for 
vagrant-test-http_example-0-100b76ee-ed70-4e15-8e52-046f992d0b4a 
I0921 19:45:38.808 [ShutdownHook, SchedulerMain:101] Stopping scheduler 
services. 
I0921 19:45:38.822 [TimeSeriesRepositoryImpl STOPPING, 
TimeSeriesRepositoryImpl:168] Variable sampler shut down 
I0921 19:45:38.839 [TearDownShutdownRegistry STOPPING, 
ShutdownRegistry$ShutdownRegistryImpl:77] Executing 4 shutdown commands. 
E0921 19:45:38.845 [TaskEventBatchWorker, BatchWorker:217] 
TaskEventBatchWorker: Failed to process batch item. Error: {} 
org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.h2.jdbc.JdbcSQLException: Database is 
already closed (to disable automatic closing at VM shutdown, add 
";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-190]
### The error may exist in org/apache/aurora/scheduler/storage/db/TaskMapper.xml
### The error may involve 
org.apache.aurora.scheduler.storage.db.TaskMapper.selectById
### The error occurred while executing a query
### SQL: SELECT       t.id AS row_id,       t.task_config_row_id AS 
task_config_row_id,       t.task_id AS task_id,       t.instance_id AS 
instance_id,       t.status AS status,       t.failure_count AS failure_count,  
     t.ancestor_task_id AS ancestor_id,       j.role AS c_j_role,       
j.environment AS c_j_environment,       j.name AS c_j_name,       h.slave_id AS 
slave_id,       h.host AS slave_host,       tp.id as tp_id,       tp.name as 
tp_name,       tp.port as tp_port,       te.id as te_id,       te.timestamp_ms 
as te_timestamp,       te.status as te_status,       te.message as te_message,  
     te.scheduler_host as te_scheduler     FROM tasks AS t     INNER JOIN 
task_configs as c ON c.id = t.task_config_row_id     INNER JOIN job_keys AS j 
ON j.id = c.job_key_id     LEFT OUTER JOIN task_ports as tp ON tp.task_row_id = 
t.id     LEFT OUTER JOIN task_events as te ON te.task_row_id = t.id     LEFT 
OUTER JOIN host_attributes AS h ON h.id = t.slave_row_id         WHERE       
t.task_id = ?
{code}


  was:
Steps to reproduce (if you are luck):
{code}
vagrant destroy
vagrant up
vagrant ssh
cd /vagrant
./build-support/release/verify-release-candidate 0.16.0-rc1
{code}

For me, this resulted in the following initial stack trace (and a couple 
different follow ups due to a corrupted DB state):

{code}
I0921 19:45:38.730 [TaskEventBatchWorker, JobUpdateControllerImpl:359] 
Forwarding task change for vagrant/test/http_example/0 
I0921 19:45:38.744 [qtp1742582311-39, Slf4jRequestLog:60] 127.0.0.1 - - 
[21/Sep/2016:19:45:38 +0000] "POST //aurora.local/api HTTP/1.1" 200 108  
I0921 19:45:38.771 [TaskEventBatchWorker, JobUpdateControllerImpl:587] 
IJobUpdateKey{job=IJobKey{role=vagrant, environment=test, name=http_example}, 
id=97ff31cc-c17f-45ef-a1d9-c16cb07388c7} evaluation result: 
EvaluationResult{status=WORKING, 
sideEffects={0=SideEffect{action=Optional.of(ADD_TASK), statusChanges=[]}}} 
I0921 19:45:38.786 [TaskEventBatchWorker, JobUpdateControllerImpl:654] 
Executing side-effects for update of IJobUpdateKey{job=IJobKey{role=vagrant, 
environment=test, name=http_example}, id=97ff31cc-c17f-45ef-a1d9-c16cb07388c7}: 
{0=SideEffect{action=Optional.of(ADD_TASK), statusChanges=[]}} 
I0921 19:45:38.789 [TaskEventBatchWorker, InstanceActionHandler$AddTask:95] 
Adding instance IInstanceKey{jobKey=IJobKey{role=vagrant, environment=test, 
name=http_example}, instanceId=0} while ROLLING_FORWARD 
I0921 19:45:38.805 [TaskEventBatchWorker, StateMachine$Builder:389] 
vagrant-test-http_example-0-100b76ee-ed70-4e15-8e52-046f992d0b4a state machine 
transition INIT -> PENDING 
I0921 19:45:38.806 [TaskEventBatchWorker, TaskStateMachine:474] Adding work 
command SAVE_STATE for 
vagrant-test-http_example-0-100b76ee-ed70-4e15-8e52-046f992d0b4a 
I0921 19:45:38.808 [ShutdownHook, SchedulerMain:101] Stopping scheduler 
services. 
I0921 19:45:38.822 [TimeSeriesRepositoryImpl STOPPING, 
TimeSeriesRepositoryImpl:168] Variable sampler shut down 
I0921 19:45:38.839 [TearDownShutdownRegistry STOPPING, 
ShutdownRegistry$ShutdownRegistryImpl:77] Executing 4 shutdown commands. 
E0921 19:45:38.845 [TaskEventBatchWorker, BatchWorker:217] 
TaskEventBatchWorker: Failed to process batch item. Error: {} 
org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.h2.jdbc.JdbcSQLException: Database is 
already closed (to disable automatic closing at VM shutdown, add 
";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-190]
### The error may exist in org/apache/aurora/scheduler/storage/db/TaskMapper.xml
### The error may involve 
org.apache.aurora.scheduler.storage.db.TaskMapper.selectById
### The error occurred while executing a query
### SQL: SELECT       t.id AS row_id,       t.task_config_row_id AS 
task_config_row_id,       t.task_id AS task_id,       t.instance_id AS 
instance_id,       t.status AS status,       t.failure_count AS failure_count,  
     t.ancestor_task_id AS ancestor_id,       j.role AS c_j_role,       
j.environment AS c_j_environment,       j.name AS c_j_name,       h.slave_id AS 
slave_id,       h.host AS slave_host,       tp.id as tp_id,       tp.name as 
tp_name,       tp.port as tp_port,       te.id as te_id,       te.timestamp_ms 
as te_timestamp,       te.status as te_status,       te.message as te_message,  
     te.scheduler_host as te_scheduler     FROM tasks AS t     INNER JOIN 
task_configs as c ON c.id = t.task_config_row_id     INNER JOIN job_keys AS j 
ON j.id = c.job_key_id     LEFT OUTER JOIN task_ports as tp ON tp.task_row_id = 
t.id     LEFT OUTER JOIN task_events as te ON te.task_row_id = t.id     LEFT 
OUTER JOIN host_attributes AS h ON h.id = t.slave_row_id         WHERE       
t.task_id = ?
{code}



> BatchWorker fails with PersistenceException
> -------------------------------------------
>
>                 Key: AURORA-1779
>                 URL: https://issues.apache.org/jira/browse/AURORA-1779
>             Project: Aurora
>          Issue Type: Bug
>            Reporter: Stephan Erb
>            Priority: Blocker
>         Attachments: aurora-scheduler.log
>
>
> Steps to reproduce (if you are lucky):
> {code}
> vagrant destroy
> vagrant up
> vagrant ssh
> cd /vagrant
> ./build-support/release/verify-release-candidate 0.16.0-rc1
> {code}
> For me, this resulted in the following initial stack trace (and a couple 
> different follow ups due to a corrupted DB state):
> {code}
> I0921 19:45:38.730 [TaskEventBatchWorker, JobUpdateControllerImpl:359] 
> Forwarding task change for vagrant/test/http_example/0 
> I0921 19:45:38.744 [qtp1742582311-39, Slf4jRequestLog:60] 127.0.0.1 - - 
> [21/Sep/2016:19:45:38 +0000] "POST //aurora.local/api HTTP/1.1" 200 108  
> I0921 19:45:38.771 [TaskEventBatchWorker, JobUpdateControllerImpl:587] 
> IJobUpdateKey{job=IJobKey{role=vagrant, environment=test, name=http_example}, 
> id=97ff31cc-c17f-45ef-a1d9-c16cb07388c7} evaluation result: 
> EvaluationResult{status=WORKING, 
> sideEffects={0=SideEffect{action=Optional.of(ADD_TASK), statusChanges=[]}}} 
> I0921 19:45:38.786 [TaskEventBatchWorker, JobUpdateControllerImpl:654] 
> Executing side-effects for update of IJobUpdateKey{job=IJobKey{role=vagrant, 
> environment=test, name=http_example}, 
> id=97ff31cc-c17f-45ef-a1d9-c16cb07388c7}: 
> {0=SideEffect{action=Optional.of(ADD_TASK), statusChanges=[]}} 
> I0921 19:45:38.789 [TaskEventBatchWorker, InstanceActionHandler$AddTask:95] 
> Adding instance IInstanceKey{jobKey=IJobKey{role=vagrant, environment=test, 
> name=http_example}, instanceId=0} while ROLLING_FORWARD 
> I0921 19:45:38.805 [TaskEventBatchWorker, StateMachine$Builder:389] 
> vagrant-test-http_example-0-100b76ee-ed70-4e15-8e52-046f992d0b4a state 
> machine transition INIT -> PENDING 
> I0921 19:45:38.806 [TaskEventBatchWorker, TaskStateMachine:474] Adding work 
> command SAVE_STATE for 
> vagrant-test-http_example-0-100b76ee-ed70-4e15-8e52-046f992d0b4a 
> I0921 19:45:38.808 [ShutdownHook, SchedulerMain:101] Stopping scheduler 
> services. 
> I0921 19:45:38.822 [TimeSeriesRepositoryImpl STOPPING, 
> TimeSeriesRepositoryImpl:168] Variable sampler shut down 
> I0921 19:45:38.839 [TearDownShutdownRegistry STOPPING, 
> ShutdownRegistry$ShutdownRegistryImpl:77] Executing 4 shutdown commands. 
> E0921 19:45:38.845 [TaskEventBatchWorker, BatchWorker:217] 
> TaskEventBatchWorker: Failed to process batch item. Error: {} 
> org.apache.ibatis.exceptions.PersistenceException: 
> ### Error querying database.  Cause: org.h2.jdbc.JdbcSQLException: Database 
> is already closed (to disable automatic closing at VM shutdown, add 
> ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-190]
> ### The error may exist in 
> org/apache/aurora/scheduler/storage/db/TaskMapper.xml
> ### The error may involve 
> org.apache.aurora.scheduler.storage.db.TaskMapper.selectById
> ### The error occurred while executing a query
> ### SQL: SELECT       t.id AS row_id,       t.task_config_row_id AS 
> task_config_row_id,       t.task_id AS task_id,       t.instance_id AS 
> instance_id,       t.status AS status,       t.failure_count AS 
> failure_count,       t.ancestor_task_id AS ancestor_id,       j.role AS 
> c_j_role,       j.environment AS c_j_environment,       j.name AS c_j_name,   
>     h.slave_id AS slave_id,       h.host AS slave_host,       tp.id as tp_id, 
>       tp.name as tp_name,       tp.port as tp_port,       te.id as te_id,     
>   te.timestamp_ms as te_timestamp,       te.status as te_status,       
> te.message as te_message,       te.scheduler_host as te_scheduler     FROM 
> tasks AS t     INNER JOIN task_configs as c ON c.id = t.task_config_row_id    
>  INNER JOIN job_keys AS j ON j.id = c.job_key_id     LEFT OUTER JOIN 
> task_ports as tp ON tp.task_row_id = t.id     LEFT OUTER JOIN task_events as 
> te ON te.task_row_id = t.id     LEFT OUTER JOIN host_attributes AS h ON h.id 
> = t.slave_row_id         WHERE       t.task_id = ?
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to