It will be hard for us to debug the problem without access to the MS SQL Server logs, but one wild idea is to decrease number of statements per transaction to decrease overall transaction size:
sqoop export -D export.statements.per.transaction=1 … I would start with 1 - if the job succeeds then we know that it’s the transaction size problem and you can tweak this number up. If the job fails even with this option, then something else is going on. > I've also tried adding -D sqoop.export.records.per.statement=500 to reduce > the load on the database, and it does not help either. Default value of this argument is 100 [1], so setting it to 500 is making the situation 5 times worst :) Jarcec Links: 1: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/AsyncSqlOutputFormat.java#L72 > On Feb 5, 2016, at 7:01 AM, Ali Thawban <orobor...@gmail.com> wrote: > > Hi, > > I'm trying to export about 300 MB of data into SQL Server (hosted in > Microsoft Azure) with --updateMode allowinsert, but am consistently getting > deadlocks at some point in the export. > > This eventually causes all attempts to fail, and the job will get back to 0% > mapping. > > I don't understand why this is happening. I'm only trying to import ~500,000 > rows. > > I've tried creating a --connection-param-file properties file with > jdbc.transaction.isolation=TRANSACTION_READ_UNCOMMITTED added, but this seems > to have no effect. > > I've also tried adding -D sqoop.export.records.per.statement=500 to reduce > the load on the database, and it does not help either. > > I've also scaled the Azure Database to the highest Premium level they have, > and I can see the CPU/throughput units (DTUs) hitting 65%, but records only > get inserted after the job fails. > > Any help is greatly appreciated!
