Re: [net] Problem with completePendingCommand and large files > 600MB
Note: curl supports an FTP range parameter.
It seems it does this by sending ABOR after the required data has arrived.
However I just tried and it does not work with the server I used.
You could try using it with your server and see if it works:
$ curl -v -r nnn-mmm -o file URL
If so you may be able to use ABOR in your app for that server.
On 25 October 2017 at 23:57, sebb wrote:
> What you are doing is stopping reading from the server before the file
> has been fully sent.
> Unless the server is expecting that, it may complain (the fact that it
> works for smaller files may be because the server has already sent the
> data).
>
> AFAICT there is no standard way to tell the server to only send a
> portion of the file.
>
> A work-round would be to fetch all the data but stop writing it to
> disk when you have got what you want.
>
> Another way might be to cancel the transfer after enough data has been
> received.
> However that relies on the server being able to process control
> channel requests whilst sending data.
>
> Or just live with the error message.
>
> It's not a bug in NET.
>
>
> On 25 October 2017 at 23:30, Bernd Eckenfels wrote:
>> 426 sounds like a server error, did you try to use a different client with
>> your server?
>>
>> Gruss
>> Bernd
>> --
>> http://bernd.eckenfels.net
>>
>> From: SeungWook Kim
>> Sent: Wednesday, October 25, 2017 6:34:02 PM
>> To: user@commons.apache.org
>> Subject: [net] Problem with completePendingCommand and large files > 600MB
>>
>> Hi,
>> I am having a problem with commons-net version* 3.6* and with large files
>>> 600MB and offset and length that does not include the whole file.
>>
>> I have included the test code below that appears to fail with large files >
>> 600 MB and an offset and length that does not include the whole file. If I
>> choose a smaller file like 90KB with offset and length, it works
>> successfully. If I choose a large file > 600MB with offset and length that
>> INCLUDES the entire file, it also works successfully. The
>> completePendingCommand replies back with "426 Failure writing network
>> stream" when the file is large > 600MB AND the offset and length is not the
>> entire file.
>>
>> I believe this is a bug but not sure. I did a quick look in the postings
>> and did not see anything that is exactly like it. Any feedback is greatly
>> appreciated.
>>
>>
>>
>> *** Need to specify serverName, userName,...absOutputFile below***
>> ***Start of Test code***
>>
>> public class FtpTest {
>> private static Logger log = LoggerFactory.getLogger(FtpTest.class);
>>
>> private static final int DEFAULT_TIMEOUT = 10*1000;
>>
>> @Before
>> public void setup() {
>>
>> }
>>
>> @Test
>> public void test() {
>> String serverName = "***";
>> String userName = "***";
>> String password = "***";
>> String absRemoteFileName = "***";
>> final long offset = ***;
>> final long length = ***;
>> String absOutputFile = "***";
>>
>> FTPClient ftpClient = new FTPClient();
>> try {
>> ftpClient.connect(serverName);
>> int reply = ftpClient.getReplyCode();
>> if (!FTPReply.isPositiveCompletion(reply)) {
>> String lastReply = ftpClient.getReplyString();
>> fail(lastReply);
>> }
>> ftpClient.addProtocolCommandListener(new
>> PrintCommandListener(new PrintWriter(System.out),
>> true));
>> ftpClient.setKeepAlive(true);
>> ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
>> //turn on SO_LINGER w/ timeout of 0 to cause connection to be
>> aborted immediately and to discard any
>> // pending data.
>> ftpClient.setSoLinger(true, 0);
>> if(ftpClient.login(userName, password)) {
>> ftpClient.enterLocalPassiveMode();
>> if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
>> {
>> log.info("Successfully connected in Passive Mode and
>> logged into {}", serverName);
>>
>> ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
>> ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
>> ftpClient.setRestartOffset(offset);
>> File writeToFile = new File(absOutputFile);
>>
>> try (InputStream inputStream =
>> ftpClient.retrieveFileStream(absRemoteFileName);
>> OutputStream outputStream =
>> FileUtils.openOutputStream(writeToFile)) {
>> if(inputStream == null || outputStream == null) {
>> fail("One of the input/output stream is null");
>> }
>> final int BUFFER_SIZE = 4096;
>> byte[] buffer = new byte[BUFFER_SIZE];
>> int bytesRead;
>>
Re: [net] Problem with completePendingCommand and large files > 600MB
What you are doing is stopping reading from the server before the file
has been fully sent.
Unless the server is expecting that, it may complain (the fact that it
works for smaller files may be because the server has already sent the
data).
AFAICT there is no standard way to tell the server to only send a
portion of the file.
A work-round would be to fetch all the data but stop writing it to
disk when you have got what you want.
Another way might be to cancel the transfer after enough data has been received.
However that relies on the server being able to process control
channel requests whilst sending data.
Or just live with the error message.
It's not a bug in NET.
On 25 October 2017 at 23:30, Bernd Eckenfels wrote:
> 426 sounds like a server error, did you try to use a different client with
> your server?
>
> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
>
> From: SeungWook Kim
> Sent: Wednesday, October 25, 2017 6:34:02 PM
> To: user@commons.apache.org
> Subject: [net] Problem with completePendingCommand and large files > 600MB
>
> Hi,
> I am having a problem with commons-net version* 3.6* and with large files
>> 600MB and offset and length that does not include the whole file.
>
> I have included the test code below that appears to fail with large files >
> 600 MB and an offset and length that does not include the whole file. If I
> choose a smaller file like 90KB with offset and length, it works
> successfully. If I choose a large file > 600MB with offset and length that
> INCLUDES the entire file, it also works successfully. The
> completePendingCommand replies back with "426 Failure writing network
> stream" when the file is large > 600MB AND the offset and length is not the
> entire file.
>
> I believe this is a bug but not sure. I did a quick look in the postings
> and did not see anything that is exactly like it. Any feedback is greatly
> appreciated.
>
>
>
> *** Need to specify serverName, userName,...absOutputFile below***
> ***Start of Test code***
>
> public class FtpTest {
> private static Logger log = LoggerFactory.getLogger(FtpTest.class);
>
> private static final int DEFAULT_TIMEOUT = 10*1000;
>
> @Before
> public void setup() {
>
> }
>
> @Test
> public void test() {
> String serverName = "***";
> String userName = "***";
> String password = "***";
> String absRemoteFileName = "***";
> final long offset = ***;
> final long length = ***;
> String absOutputFile = "***";
>
> FTPClient ftpClient = new FTPClient();
> try {
> ftpClient.connect(serverName);
> int reply = ftpClient.getReplyCode();
> if (!FTPReply.isPositiveCompletion(reply)) {
> String lastReply = ftpClient.getReplyString();
> fail(lastReply);
> }
> ftpClient.addProtocolCommandListener(new
> PrintCommandListener(new PrintWriter(System.out),
> true));
> ftpClient.setKeepAlive(true);
> ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
> //turn on SO_LINGER w/ timeout of 0 to cause connection to be
> aborted immediately and to discard any
> // pending data.
> ftpClient.setSoLinger(true, 0);
> if(ftpClient.login(userName, password)) {
> ftpClient.enterLocalPassiveMode();
> if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
> {
> log.info("Successfully connected in Passive Mode and
> logged into {}", serverName);
>
> ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
> ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
> ftpClient.setRestartOffset(offset);
> File writeToFile = new File(absOutputFile);
>
> try (InputStream inputStream =
> ftpClient.retrieveFileStream(absRemoteFileName);
> OutputStream outputStream =
> FileUtils.openOutputStream(writeToFile)) {
> if(inputStream == null || outputStream == null) {
> fail("One of the input/output stream is null");
> }
> final int BUFFER_SIZE = 4096;
> byte[] buffer = new byte[BUFFER_SIZE];
> int bytesRead;
> int len = (int)length;
> int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
> : len;
> while ((bytesRead = inputStream.read(buffer, 0,
> bufferSize)) != -1) {
> len -= bytesRead;
> bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
> len;
>
> outputStream.write(buffer, 0, bytesRead);
>
> if(len <= 0) {
> break;
Re: [net] Problem with completePendingCommand and large files > 600MB
426 sounds like a server error, did you try to use a different client with your
server?
Gruss
Bernd
--
http://bernd.eckenfels.net
From: SeungWook Kim
Sent: Wednesday, October 25, 2017 6:34:02 PM
To: user@commons.apache.org
Subject: [net] Problem with completePendingCommand and large files > 600MB
Hi,
I am having a problem with commons-net version* 3.6* and with large files
> 600MB and offset and length that does not include the whole file.
I have included the test code below that appears to fail with large files >
600 MB and an offset and length that does not include the whole file. If I
choose a smaller file like 90KB with offset and length, it works
successfully. If I choose a large file > 600MB with offset and length that
INCLUDES the entire file, it also works successfully. The
completePendingCommand replies back with "426 Failure writing network
stream" when the file is large > 600MB AND the offset and length is not the
entire file.
I believe this is a bug but not sure. I did a quick look in the postings
and did not see anything that is exactly like it. Any feedback is greatly
appreciated.
*** Need to specify serverName, userName,...absOutputFile below***
***Start of Test code***
public class FtpTest {
private static Logger log = LoggerFactory.getLogger(FtpTest.class);
private static final int DEFAULT_TIMEOUT = 10*1000;
@Before
public void setup() {
}
@Test
public void test() {
String serverName = "***";
String userName = "***";
String password = "***";
String absRemoteFileName = "***";
final long offset = ***;
final long length = ***;
String absOutputFile = "***";
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(serverName);
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
String lastReply = ftpClient.getReplyString();
fail(lastReply);
}
ftpClient.addProtocolCommandListener(new
PrintCommandListener(new PrintWriter(System.out),
true));
ftpClient.setKeepAlive(true);
ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
//turn on SO_LINGER w/ timeout of 0 to cause connection to be
aborted immediately and to discard any
// pending data.
ftpClient.setSoLinger(true, 0);
if(ftpClient.login(userName, password)) {
ftpClient.enterLocalPassiveMode();
if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
{
log.info("Successfully connected in Passive Mode and
logged into {}", serverName);
ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
ftpClient.setRestartOffset(offset);
File writeToFile = new File(absOutputFile);
try (InputStream inputStream =
ftpClient.retrieveFileStream(absRemoteFileName);
OutputStream outputStream =
FileUtils.openOutputStream(writeToFile)) {
if(inputStream == null || outputStream == null) {
fail("One of the input/output stream is null");
}
final int BUFFER_SIZE = 4096;
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
int len = (int)length;
int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
: len;
while ((bytesRead = inputStream.read(buffer, 0,
bufferSize)) != -1) {
len -= bytesRead;
bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
len;
outputStream.write(buffer, 0, bytesRead);
if(len <= 0) {
break;
}
}
} finally {
if(ftpClient.completePendingCommand()) {
log.info("Successful complete pending command");
} else {
fail("Unsuccessful complete pending command");
}
}
} else {
fail("Problem entering passive mode. All modern ftp
server supports passive mode, but " +
serverName + " doesn't??");
}
} else {
fail("Bad credentials to server " + serverName);
}
} catch (Exception e) {
fail(e.toString() + "\n" + e.getStackTrace().toString());
}
}
}
***End of Test code***
Thank you
-Seung
[net] Problem with completePendingCommand and large files > 600MB
Hi,
I am having a problem with commons-net version* 3.6* and with large files
> 600MB and offset and length that does not include the whole file.
I have included the test code below that appears to fail with large files >
600 MB and an offset and length that does not include the whole file. If I
choose a smaller file like 90KB with offset and length, it works
successfully. If I choose a large file > 600MB with offset and length that
INCLUDES the entire file, it also works successfully. The
completePendingCommand replies back with "426 Failure writing network
stream" when the file is large > 600MB AND the offset and length is not the
entire file.
I believe this is a bug but not sure. I did a quick look in the postings
and did not see anything that is exactly like it. Any feedback is greatly
appreciated.
*** Need to specify serverName, userName,...absOutputFile below***
***Start of Test code***
public class FtpTest {
private static Logger log = LoggerFactory.getLogger(FtpTest.class);
private static final int DEFAULT_TIMEOUT = 10*1000;
@Before
public void setup() {
}
@Test
public void test() {
String serverName = "***";
String userName = "***";
String password = "***";
String absRemoteFileName = "***";
final long offset = ***;
final long length = ***;
String absOutputFile = "***";
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(serverName);
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
String lastReply = ftpClient.getReplyString();
fail(lastReply);
}
ftpClient.addProtocolCommandListener(new
PrintCommandListener(new PrintWriter(System.out),
true));
ftpClient.setKeepAlive(true);
ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
//turn on SO_LINGER w/ timeout of 0 to cause connection to be
aborted immediately and to discard any
// pending data.
ftpClient.setSoLinger(true, 0);
if(ftpClient.login(userName, password)) {
ftpClient.enterLocalPassiveMode();
if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
{
log.info("Successfully connected in Passive Mode and
logged into {}", serverName);
ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
ftpClient.setRestartOffset(offset);
File writeToFile = new File(absOutputFile);
try (InputStream inputStream =
ftpClient.retrieveFileStream(absRemoteFileName);
OutputStream outputStream =
FileUtils.openOutputStream(writeToFile)) {
if(inputStream == null || outputStream == null) {
fail("One of the input/output stream is null");
}
final int BUFFER_SIZE = 4096;
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
int len = (int)length;
int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
: len;
while ((bytesRead = inputStream.read(buffer, 0,
bufferSize)) != -1) {
len -= bytesRead;
bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
len;
outputStream.write(buffer, 0, bytesRead);
if(len <= 0) {
break;
}
}
} finally {
if(ftpClient.completePendingCommand()) {
log.info("Successful complete pending command");
} else {
fail("Unsuccessful complete pending command");
}
}
} else {
fail("Problem entering passive mode. All modern ftp
server supports passive mode, but " +
serverName + " doesn't??");
}
} else {
fail("Bad credentials to server " + serverName);
}
} catch (Exception e) {
fail(e.toString() + "\n" + e.getStackTrace().toString());
}
}
}
***End of Test code***
Thank you
-Seung
[dbcp] Narayana with DBCP using XADataSource
Hello, I am trying to use Narayana transaction manager with Apache Commons DBCP. The issue is that I would like to use XA DataSource and there is a very small amount of documentation about how to set DBCP with XA. From what I read I understood that I should probably use BasicManagedDataSource instead of BasicDataSource, which is only for non-XA connections, right? The other thing is that now I have 3 options according to https://github.com/apache/commons-dbcp/blob/master/src/main/java/org/apache/commons/dbcp2/managed/BasicManagedDataSource.java#L135 1. Use ordinary driverClassName from BasicDataSource - it will create non-XA connection factory which is then wrapped by xaConnectionFactory. 2. Set xaDataSource which is of type String - so I will provide e.g. oracle.jdbc.xa.client.OracleXADataSource and then if I haven't created instance myself it will be created for me inside createConnectionFactory method. Here the biggest question is - How then I provide/set e.g. URL to the data source if it is created without any additional calls in the middle of the initialization of whole DBCP - I cannot call getXaDataSourceInstance and call setUrl myself because the DBCP will have been already initialized by then. User and password are obtained via getUser/getPassword methods, so this is OK. GetUrl method is used only inside BasicDataSource and it is not used on xaDataSource instance. 3. Create oracle.jdbc.xa.client.OracleXADataSource myself, call methods I want to call on it and set it on the BasicManagedDataSource. After that I guess I just need to set Narayana TransactionManager via BasicManagedDataSource.setTransactionManager method and I should be good to go, right? Thank you very much for answers. The biggest question for me is the point no. 2 - if DBCP creates XADataSource for me automatically - where does it set the URL, driverType and other properties in case they are needed? At least the URL is mandatory. Best regards, Marian Macik - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
