Github user lresende commented on the pull request:
https://github.com/apache/spark/pull/9893#issuecomment-174039005
@JoshRosen For this PR to work, there is a need to manually deploy the JDBC
driver as described on the pom. This is similar to the requirement that in
order to run these tests, people must have docker installed.
As for deploying the driver into maven or something, I am trying to get
that approved, but for now people can download the driver manually via the link
described in the pom.
As for running these, I have ran it on our Jenkins environment, and see the
results of the DB2 Integration tests below.
Could you please review this, help with the Jenkins slaves configuration,
and merge so we can move forward with the rest of the DB2 dialect prs waiting
on this.
Db2IntegrationSuite:
16/01/22 12:26:56 INFO Slf4jLogger: Slf4jLogger started
16/01/22 12:26:56 INFO Remoting: Starting remoting
16/01/22 12:26:56 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriverActorSystem@localhost:33460]
16/01/22 12:26:56 INFO Utils: Successfully started service
'sparkDriverActorSystem' on port 33460.
16/01/22 12:26:56 INFO SparkEnv: Registering MapOutputTracker
16/01/22 12:26:56 INFO SparkEnv: Registering BlockManagerMaster
16/01/22 12:26:56 INFO DiskBlockManager: Created local directory at
/a/workspace/Spark-Build_and_Test_with_docker/docker-integration-tests/target/tmp/blockmgr-73d7995e-9327-46d8-9a4e-3a9b991e2782
16/01/22 12:26:56 INFO MemoryStore: MemoryStore started with capacity 2.0 GB
16/01/22 12:26:56 INFO SparkEnv: Registering OutputCommitCoordinator
16/01/22 12:26:56 INFO Executor: Starting executor ID driver on host
localhost
16/01/22 12:26:56 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 36866.
16/01/22 12:26:56 INFO NettyBlockTransferService: Server created on 36866
16/01/22 12:26:56 INFO BlockManagerMaster: Trying to register BlockManager
16/01/22 12:26:56 INFO BlockManagerMasterEndpoint: Registering block
manager localhost:36866 with 2.0 GB RAM, BlockManagerId(driver, localhost,
36866)
16/01/22 12:26:56 INFO BlockManagerMaster: Registered BlockManager
16/01/22 12:26:56 INFO DefaultDockerClient: Creating container with
ContainerConfig: ContainerConfig{hostname=null, domainname=null, username=null,
attachStdin=null, attachStdout=null, attachStderr=null, portSpecs=null,
exposedPorts=[50000/tcp], tty=null, openStdin=null, stdinOnce=null,
env=[DB2INST1_PASSWORD=rootpass, LICENSE=accept], cmd=[db2start],
image=lresende/db2express-c:10.5.0.5-3.10.0, volumes=null, workingDir=null,
entrypoint=null, networkDisabled=false, onBuild=null, labels=null,
macAddress=null, hostConfig=HostConfig{binds=null, containerIDFile=null,
lxcConf=null, privileged=null,
portBindings={50000/tcp=[PortBinding{hostIp=9.30.122.152, hostPort=51408}]},
links=null, publishAllPorts=null, dns=null, dnsSearch=null, extraHosts=null,
volumesFrom=null, networkMode=bridge, securityOpt=null, memory=null,
memorySwap=null, cpuShares=null, cpusetCpus=null, cpuQuota=null,
cgroupParent=null}}
16/01/22 12:26:58 INFO DefaultDockerClient: Starting container with Id:
bc6620ff6e3c41521e48752164ded0290e4d97c30ce1c30d68436fa45ea6863c
16/01/22 12:27:05 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'Basic test' =====
16/01/22 12:27:05 INFO SparkContext: Starting job: apply at
Transformer.scala:22
16/01/22 12:27:05 INFO DAGScheduler: Got job 0 (apply at
Transformer.scala:22) with 1 output partitions
16/01/22 12:27:05 INFO DAGScheduler: Final stage: ResultStage 0 (apply at
Transformer.scala:22)
16/01/22 12:27:05 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:05 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:05 INFO DAGScheduler: Submitting ResultStage 0
(MapPartitionsRDD[2] at apply at Transformer.scala:22), which has no missing
parents
16/01/22 12:27:05 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 6.6 KB, free 6.6 KB)
16/01/22 12:27:05 INFO MemoryStore: Block broadcast_0_piece0 stored as
bytes in memory (estimated size 3.3 KB, free 9.9 KB)
16/01/22 12:27:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on localhost:36866 (size: 3.3 KB, free: 2.0 GB)
16/01/22 12:27:05 INFO SparkContext: Created broadcast 0 from broadcast at
DAGScheduler.scala:1010
16/01/22 12:27:05 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 0 (MapPartitionsRDD[2] at apply at Transformer.scala:22)
16/01/22 12:27:05 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/01/22 12:27:05 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
0, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:05 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/01/22 12:27:05 INFO JDBCRDD: closed connection
16/01/22 12:27:05 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0).
1476 bytes result sent to driver
16/01/22 12:27:05 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID
0) in 289 ms on localhost (1/1)
16/01/22 12:27:05 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
16/01/22 12:27:05 INFO DAGScheduler: ResultStage 0 (apply at
Transformer.scala:22) finished in 0.290 s
16/01/22 12:27:05 INFO DAGScheduler: Job 0 finished: apply at
Transformer.scala:22, took 0.299543 s
16/01/22 12:27:05 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'Basic test' =====
16/01/22 12:27:05 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'Numeric types'
=====
- Basic test
16/01/22 12:27:06 INFO SparkContext: Starting job: apply at
Transformer.scala:22
16/01/22 12:27:06 INFO DAGScheduler: Got job 1 (apply at
Transformer.scala:22) with 1 output partitions
16/01/22 12:27:06 INFO DAGScheduler: Final stage: ResultStage 1 (apply at
Transformer.scala:22)
16/01/22 12:27:06 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:06 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:06 INFO DAGScheduler: Submitting ResultStage 1
(MapPartitionsRDD[5] at apply at Transformer.scala:22), which has no missing
parents
16/01/22 12:27:06 INFO MemoryStore: Block broadcast_1 stored as values in
memory (estimated size 7.4 KB, free 17.3 KB)
16/01/22 12:27:06 INFO MemoryStore: Block broadcast_1_piece0 stored as
bytes in memory (estimated size 3.7 KB, free 21.0 KB)
16/01/22 12:27:06 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory
on localhost:36866 (size: 3.7 KB, free: 2.0 GB)
16/01/22 12:27:06 INFO SparkContext: Created broadcast 1 from broadcast at
DAGScheduler.scala:1010
16/01/22 12:27:06 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 1 (MapPartitionsRDD[5] at apply at Transformer.scala:22)
16/01/22 12:27:06 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/01/22 12:27:06 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID
1, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:06 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
16/01/22 12:27:06 INFO CodeGenerator: Code generated in 12.625054 ms
16/01/22 12:27:06 INFO JDBCRDD: closed connection
16/01/22 12:27:06 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1).
1467 bytes result sent to driver
16/01/22 12:27:06 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID
1) in 392 ms on localhost (1/1)
16/01/22 12:27:06 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks
have all completed, from pool
16/01/22 12:27:06 INFO DAGScheduler: ResultStage 1 (apply at
Transformer.scala:22) finished in 0.393 s
16/01/22 12:27:06 INFO DAGScheduler: Job 1 finished: apply at
Transformer.scala:22, took 0.400140 s
16/01/22 12:27:06 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'Numeric types' =====
- Numeric types
16/01/22 12:27:06 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'Date types' =====
16/01/22 12:27:06 INFO SparkContext: Starting job: apply at
Transformer.scala:22
16/01/22 12:27:06 INFO DAGScheduler: Got job 2 (apply at
Transformer.scala:22) with 1 output partitions
16/01/22 12:27:06 INFO DAGScheduler: Final stage: ResultStage 2 (apply at
Transformer.scala:22)
16/01/22 12:27:06 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:06 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:06 INFO DAGScheduler: Submitting ResultStage 2
(MapPartitionsRDD[8] at apply at Transformer.scala:22), which has no missing
parents
16/01/22 12:27:06 INFO MemoryStore: Block broadcast_2 stored as values in
memory (estimated size 6.7 KB, free 27.6 KB)
16/01/22 12:27:06 INFO MemoryStore: Block broadcast_2_piece0 stored as
bytes in memory (estimated size 3.4 KB, free 31.0 KB)
16/01/22 12:27:06 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory
on localhost:36866 (size: 3.4 KB, free: 2.0 GB)
16/01/22 12:27:06 INFO SparkContext: Created broadcast 2 from broadcast at
DAGScheduler.scala:1010
16/01/22 12:27:06 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 2 (MapPartitionsRDD[8] at apply at Transformer.scala:22)
16/01/22 12:27:06 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
16/01/22 12:27:06 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID
2, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:06 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
16/01/22 12:27:07 INFO CodeGenerator: Code generated in 11.399165 ms
16/01/22 12:27:07 INFO JDBCRDD: closed connection
16/01/22 12:27:07 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2).
1427 bytes result sent to driver
16/01/22 12:27:07 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID
2) in 315 ms on localhost (1/1)
16/01/22 12:27:07 INFO DAGScheduler: ResultStage 2 (apply at
Transformer.scala:22) finished in 0.315 s
16/01/22 12:27:07 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks
have all completed, from pool
16/01/22 12:27:07 INFO DAGScheduler: Job 2 finished: apply at
Transformer.scala:22, took 0.321779 s
16/01/22 12:27:07 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'Date types' =====
- Date types
16/01/22 12:27:07 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'String types'
=====
16/01/22 12:27:07 INFO SparkContext: Starting job: apply at
Transformer.scala:22
16/01/22 12:27:07 INFO DAGScheduler: Got job 3 (apply at
Transformer.scala:22) with 1 output partitions
16/01/22 12:27:07 INFO DAGScheduler: Final stage: ResultStage 3 (apply at
Transformer.scala:22)
16/01/22 12:27:07 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:07 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:07 INFO DAGScheduler: Submitting ResultStage 3
(MapPartitionsRDD[11] at apply at Transformer.scala:22), which has no missing
parents
16/01/22 12:27:07 INFO MemoryStore: Block broadcast_3 stored as values in
memory (estimated size 6.7 KB, free 37.7 KB)
16/01/22 12:27:07 INFO MemoryStore: Block broadcast_3_piece0 stored as
bytes in memory (estimated size 3.4 KB, free 41.1 KB)
16/01/22 12:27:07 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory
on localhost:36866 (size: 3.4 KB, free: 2.0 GB)
16/01/22 12:27:07 INFO SparkContext: Created broadcast 3 from broadcast at
DAGScheduler.scala:1010
16/01/22 12:27:07 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 3 (MapPartitionsRDD[11] at apply at Transformer.scala:22)
16/01/22 12:27:07 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
16/01/22 12:27:07 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID
3, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:07 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
16/01/22 12:27:07 INFO CodeGenerator: Code generated in 8.298278 ms
16/01/22 12:27:07 INFO JDBCRDD: closed connection
16/01/22 12:27:07 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3).
1475 bytes result sent to driver
16/01/22 12:27:07 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID
3) in 360 ms on localhost (1/1)
16/01/22 12:27:07 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks
have all completed, from pool
16/01/22 12:27:07 INFO DAGScheduler: ResultStage 3 (apply at
Transformer.scala:22) finished in 0.361 s
16/01/22 12:27:07 INFO DAGScheduler: Job 3 finished: apply at
Transformer.scala:22, took 0.368710 s
16/01/22 12:27:07 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'String types' =====
- String types
16/01/22 12:27:07 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'Basic write
test' =====
16/01/22 12:27:09 INFO SparkContext: Starting job: apply at
Transformer.scala:22
16/01/22 12:27:09 INFO DAGScheduler: Got job 4 (apply at
Transformer.scala:22) with 1 output partitions
16/01/22 12:27:09 INFO DAGScheduler: Final stage: ResultStage 4 (apply at
Transformer.scala:22)
16/01/22 12:27:09 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:09 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:09 INFO DAGScheduler: Submitting ResultStage 4
(MapPartitionsRDD[14] at apply at Transformer.scala:22), which has no missing
parents
16/01/22 12:27:09 INFO MemoryStore: Block broadcast_4 stored as values in
memory (estimated size 7.1 KB, free 48.2 KB)
16/01/22 12:27:09 INFO MemoryStore: Block broadcast_4_piece0 stored as
bytes in memory (estimated size 3.6 KB, free 51.9 KB)
16/01/22 12:27:09 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory
on localhost:36866 (size: 3.6 KB, free: 2.0 GB)
16/01/22 12:27:09 INFO SparkContext: Created broadcast 4 from broadcast at
DAGScheduler.scala:1010
16/01/22 12:27:09 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 4 (MapPartitionsRDD[14] at apply at Transformer.scala:22)
16/01/22 12:27:09 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
16/01/22 12:27:09 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID
4, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:09 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
16/01/22 12:27:09 INFO JDBCRDD: closed connection
16/01/22 12:27:09 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4).
1165 bytes result sent to driver
16/01/22 12:27:09 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID
4) in 387 ms on localhost (1/1)
16/01/22 12:27:09 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks
have all completed, from pool
16/01/22 12:27:09 INFO DAGScheduler: ResultStage 4 (apply at
Transformer.scala:22) finished in 0.388 s
16/01/22 12:27:09 INFO DAGScheduler: Job 4 finished: apply at
Transformer.scala:22, took 0.394228 s
16/01/22 12:27:10 INFO SparkContext: Starting job: apply at
Transformer.scala:22
16/01/22 12:27:10 INFO DAGScheduler: Got job 5 (apply at
Transformer.scala:22) with 1 output partitions
16/01/22 12:27:10 INFO DAGScheduler: Final stage: ResultStage 5 (apply at
Transformer.scala:22)
16/01/22 12:27:10 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:10 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:10 INFO DAGScheduler: Submitting ResultStage 5
(MapPartitionsRDD[17] at apply at Transformer.scala:22), which has no missing
parents
16/01/22 12:27:10 INFO MemoryStore: Block broadcast_5 stored as values in
memory (estimated size 7.2 KB, free 59.1 KB)
16/01/22 12:27:10 INFO MemoryStore: Block broadcast_5_piece0 stored as
bytes in memory (estimated size 3.7 KB, free 62.8 KB)
16/01/22 12:27:10 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory
on localhost:36866 (size: 3.7 KB, free: 2.0 GB)
16/01/22 12:27:10 INFO SparkContext: Created broadcast 5 from broadcast at
DAGScheduler.scala:1010
16/01/22 12:27:10 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 5 (MapPartitionsRDD[17] at apply at Transformer.scala:22)
16/01/22 12:27:10 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
16/01/22 12:27:10 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID
5, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:10 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
16/01/22 12:27:10 INFO JDBCRDD: closed connection
16/01/22 12:27:10 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5).
1165 bytes result sent to driver
16/01/22 12:27:10 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID
5) in 433 ms on localhost (1/1)
16/01/22 12:27:10 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks
have all completed, from pool
16/01/22 12:27:10 INFO DAGScheduler: ResultStage 5 (apply at
Transformer.scala:22) finished in 0.435 s
16/01/22 12:27:10 INFO DAGScheduler: Job 5 finished: apply at
Transformer.scala:22, took 0.441109 s
16/01/22 12:27:10 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'Basic write test' =====
- Basic write test
16/01/22 12:27:16 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
16/01/22 12:27:16 INFO MemoryStore: MemoryStore cleared
16/01/22 12:27:16 INFO BlockManager: BlockManager stopped
16/01/22 12:27:16 INFO BlockManagerMaster: BlockManagerMaster stopped
16/01/22 12:27:16 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
16/01/22 12:27:16 INFO RemoteActorRefProvider$RemotingTerminator: Shutting
down remote daemon.
16/01/22 12:27:16 INFO RemoteActorRefProvider$RemotingTerminator: Remote
daemon shut down; proceeding with flushing remote transports.
16/01/22 12:27:16 INFO SparkContext: Successfully stopped SparkContext
16/01/22 12:27:16 INFO RemoteActorRefProvider$RemotingTerminator: Remoting
shut down.
Run completed in 58 seconds, 288 milliseconds.
Total number of tests run: 12
Suites: completed 4, aborted 0
Tests: succeeded 12, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
